Text-based Editing of Talking-head Video

Ohad Fried, Ayush Tewari, Michael Zollhöfer, Adam Finkelstein, Eli Shechtman, Dan B. Goldman, Kyle Genova, Zeyu Jin, Christian Theobalt, Maneesh Agrawala

Research output: Contribution to journalArticle

8 Scopus citations

Abstract

Editing talking-head video to change the speech content or to remove filler words is challenging. We propose a novel method to edit talking-head video based on its transcript to produce a realistic output video in which the dialogue of the speaker has been modified, while maintaining a seamless audio-visual flow (i.e. no jump cuts). Our method automatically annotates an input talking-head video with phonemes, visemes, 3D face pose and geometry, reflectance, expression and scene illumination per frame. To edit a video, the user has to only edit the transcript, and an optimization strategy then chooses segments of the input corpus as base material. The annotated parameters corresponding to the selected segments are seamlessly stitched together and used to produce an intermediate video representation in which the lower half of the face is rendered with a parametric face model. Finally, a recurrent video generation network transforms this representation to a photorealistic video that matches the edited transcript. We demonstrate a large variety of edits, such as the addition, removal, and alteration of words, as well as convincing language translation and full sentence synthesis.

Original languageEnglish (US)
Article number68
JournalACM Transactions on Graphics
Volume38
Issue number4
DOIs
StatePublished - Jul 2019

All Science Journal Classification (ASJC) codes

  • Computer Graphics and Computer-Aided Design

Keywords

  • Dubbing
  • Face parameterization
  • Face tracking
  • Neural rendering
  • Talking heads
  • Text-based video editing
  • Visemes

Fingerprint Dive into the research topics of 'Text-based Editing of Talking-head Video'. Together they form a unique fingerprint.

  • Cite this

    Fried, O., Tewari, A., Zollhöfer, M., Finkelstein, A., Shechtman, E., Goldman, D. B., Genova, K., Jin, Z., Theobalt, C., & Agrawala, M. (2019). Text-based Editing of Talking-head Video. ACM Transactions on Graphics, 38(4), [68]. https://doi.org/10.1145/3306346.3323028