Asynchronous Pipeline Parallelism for Real-Time Multilingual Lip Synchronization in Video Communication Systems

arXiv:2512.18318 · cs.MM, cs.AI, cs.CV, cs.DC, cs.NI · Submitted 2025-12-20 · Read on arXiv

cs.MM, cs.AI, cs.CV, cs.DC, cs.NI

Submitted: 2025-12-20

Updated: 2025-12-20

Comments: Accepted to IEEE Big Data 2025, AIDE4IoT Workshop. Copyright \c{opyright} 2025 IEEE

DOI: 10.1109/BigData66926.2025.11400775

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Accuracy — 11× in this paper · explained in 1 episode
• Alignment — 10× in this paper · explained in 1 episode
• Transformer — 9× in this paper · explained in 2 episodes
• Machine Translation — 8× in this paper
• Decoupling — 6× in this paper · explained in 2 episodes
• Core — 5× in this paper · explained in 2 episodes
• Quantization — 5× in this paper · explained in 12 episodes
• Consistency — 3× in this paper · explained in 3 episodes
• Ablation — 2× in this paper · explained in 1 episode
• Diffusion Model — 2× in this paper · explained in 8 episodes
• Diffusion Models — 2× in this paper · explained in 6 episodes
• Fidelity — 2× in this paper · explained in 2 episodes
• Generative AI — 2× in this paper · explained in 2 episodes
• Generative Model — 2× in this paper · explained in 2 episodes
• Generative Models — 2× in this paper · explained in 2 episodes
• Modularity — 2× in this paper · explained in 9 episodes
• Real-Time Visual Pipeline — 2× in this paper
• Actor — 1× in this paper · explained in 3 episodes
• Adaptability — 1× in this paper · explained in 1 episode
• Artificial Intelligence of Things — 1× in this paper
• Asynchronous Translation Pipeline — 1× in this paper
• Attention Mechanism — 1× in this paper · explained in 3 episodes
• Automatic Speech Recognition — 1× in this paper
• Benchmarking — 1× in this paper · explained in 3 episodes
• Calibration — 1× in this paper · explained in 7 episodes
• Digital Object Identifier — 1× in this paper
• Edge Devices — 1× in this paper · explained in 2 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Generative Modeling — 1× in this paper · explained in 3 episodes
• Interpretability — 1× in this paper · explained in 15 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• M EAN O PINION S CORES — 1× in this paper
• Modular Architecture — 1× in this paper · explained in 4 episodes
• Resource Constraints — 1× in this paper · explained in 3 episodes

Sources

• A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild
• Listen, Attend and Spell
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Sequence to Sequence Learning with Neural Networks
• Effective Approaches to Attention-based Neural Machine Translation
• Tacotron: Towards End-to-End Speech Synthesis
• FastSpeech: Fast, Robust and Controllable Text to Speech
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone
• AudioPaLM: A Large Language Model That Can Speak and Listen
• Diff2Lip: Audio Conditioned Diffusion Models for Lip-Synchronization
• LatentSync: Taming Audio-Conditioned Latent Diffusion Models for Lip Sync with SyncNet Supervision
• AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis
• SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis
• A Bridge from Audio to Video: Phoneme-Viseme Alignment Allows Every Face to Speak Multiple Languages
• S$^3$FD: Single Shot Scale-invariant Face Detector

Related papers

• ControlFoley: Unified and Controllable Video-to-Audio Generation with Cross-Modal Conflict Handling
• Zero-shot Video Moment Retrieval via Off-the-shelf Multimodal Large Language Models
• Mitigating GenAI-powered Evidence Pollution for Out-of-Context Multimodal Misinformation Detection
• A Rate-Distortion-Classification Approach for Lossy Image Compression