SyncVoice: Simple and Effective Automatic Video Dubbing with Vision-Augmented TTS

arXiv:2512.05126 · eess.AS, cs.AI, cs.CL, cs.CV, cs.MM, cs.SD · Submitted 2025-11-23 · Read on arXiv

eess.AS, cs.AI, cs.CL, cs.CV, cs.MM, cs.SD

Submitted: 2025-11-23

Updated: 2026-09-15

Code: https://github.com/wenet-e2e/wespeaker

Project page: https://wkd88.github.io/syncvoice

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Consistency — 5× in this paper · explained in 3 episodes
• Fidelity — 5× in this paper · explained in 2 episodes
• Ablation — 4× in this paper · explained in 1 episode
• Embeddings — 4× in this paper · explained in 2 episodes
• Fine-Tuning — 4× in this paper · explained in 5 episodes
• Flow Matching — 4× in this paper · explained in 8 episodes
• Conference on Computer Vision and Pattern Recognition — 3× in this paper
• Verification — 3× in this paper · explained in 2 episodes
• Accuracy — 2× in this paper · explained in 1 episode
• Ablation Studies — 1× in this paper · explained in 1 episode
• Ablation study — 1× in this paper · explained in 1 episode
• Alignment — 1× in this paper · explained in 1 episode
• Cosine Similarity — 1× in this paper · explained in 3 episodes
• Diffusion Model — 1× in this paper · explained in 8 episodes
• Diffusion Models — 1× in this paper · explained in 6 episodes
• Generative Model — 1× in this paper · explained in 2 episodes
• Generative Models — 1× in this paper · explained in 3 episodes
• Latent Diffusion Model — 1× in this paper · explained in 2 episodes
• Latent Diffusion Models — 1× in this paper · explained in 2 episodes
• Latent Space — 1× in this paper · explained in 7 episodes
• Transformer — 1× in this paper · explained in 2 episodes
• Word Error Rate — 1× in this paper

Sources

• VoiceCraft-Dub: Automated Video Dubbing with Neural Codec Language Models
• F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching
• IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speech
• ZipVoice: Fast and High-Quality Zero-Shot Text-to-Speech with Flow Matching
• MiniMax-Speech: Intrinsic Zero-Shot Text-to-Speech with a Learnable Speaker Encoder
• LatentSync: Taming Audio-Conditioned Latent Diffusion Models for Lip Sync with SyncNet Supervision

Related papers

• X-VC: Zero-shot Streaming Voice Conversion in Codec Space
• Autoregressive Guidance of Deep Spatially Selective Filters using Bayesian Tracking for Efficient Extraction of Moving Speakers
• Anonymization, Not Elimination: Utility-Preserved Speech Anonymization
• Towards Audio Token Compression in Large Audio Language Models
• WaveScat: Wavelet Scattering Front-Ends with Self-Supervised Features for Speech Deepfake Detection
• ProPS: Prompted Profile Synthesis for Natural Language-Conditioned Speaker Embedding Distributions