VoCo: Text-based insertion and replacement in audio narration

Zeyu Jin, Gautham J. Mysore, Stephen Diverdi, Jingwan Lu, Adam Finkelstein

Research output: Contribution to journalConference articlepeer-review

45 Scopus citations

Abstract

Editing audio narration using conventional software typically involves many painstaking low-level manipulations. Some state of the art systems allow the editor to work in a text transcript of the narration, and perform select, cut, copy and paste operations directly in the transcript; these operations are then automatically applied to the waveform in a straightforward manner. However, an obvious gap in the text-based interface is the ability to type new words not appearing in the transcript, for example inserting a new word for emphasis or replacing a misspoken word. While high-quality voice synthesizers exist today, the challenge is to synthesize the new word in a voice that matches the rest of the narration. This paper presents a system that can synthesize a new word or short phrase such that it blends seamlessly in the context of the existing narration. Our approach is to use a text to speech synthesizer to say the word in a generic voice, and then use voice conversion to convert it into a voice that matches the narration. Offering a range of degrees of control to the editor, our interface supports fully automatic synthesis, selection among a candidate set of alternative pronunciations, fine control over edit placements and pitch profiles, and even guidance by the editors own voice. The paper presents studies showing that the output of our method is preferred over baseline methods and often indistinguishable from the original voice.

Original languageEnglish (US)
Article number96
JournalACM Transactions on Graphics
Volume36
Issue number4
DOIs
StatePublished - 2017
EventACM SIGGRAPH 2017 - Los Angeles, United States
Duration: Jul 30 2017Aug 3 2017

All Science Journal Classification (ASJC) codes

  • Computer Graphics and Computer-Aided Design

Keywords

  • Audio
  • Human computer interaction

Fingerprint

Dive into the research topics of 'VoCo: Text-based insertion and replacement in audio narration'. Together they form a unique fingerprint.

Cite this