Improving Audiovisual Speech Recognition through Synthetic Visual Data Augmentation

arXiv:2609.31961 · eess.AS, cs.CL, cs.SD, eess.IV · Submitted 2026-09-25 · Read on arXiv

eess.AS, cs.CL, cs.SD, eess.IV

Submitted: 2026-09-25

Updated: 2026-09-25

Code: https://github.com/Pol-Buitrago/SynthAVSR

Terminology

• Benchmark — 23× in this paper · explained in 2 episodes
• Data Augmentation — 16× in this paper · explained in 7 episodes
• Robustness — 9× in this paper · explained in 5 episodes
• Fine-Tuning — 6× in this paper · explained in 5 episodes
• Visual Speech Recognition — 5× in this paper
• Word Error Rate — 4× in this paper
• Accuracy — 3× in this paper · explained in 1 episode
• Integrated Gradients — 3× in this paper
• Adam — 2× in this paper · explained in 2 episodes
• Empirical Methods in Natural Language Processing — 2× in this paper
• International Conference on Learning Representations — 2× in this paper
• Low-Resource Languages — 2× in this paper · explained in 4 episodes
• Synthetic Data Generation — 2× in this paper · explained in 5 episodes
• Transformer — 2× in this paper · explained in 2 episodes
• Adaptability — 1× in this paper · explained in 1 episode
• Alignment — 1× in this paper · explained in 1 episode
• Barcelona Supercomputing Center — 1× in this paper
• Conference on Computer Vision and Pattern Recognition — 1× in this paper
• Consistency — 1× in this paper · explained in 3 episodes
• Domain Shift — 1× in this paper · explained in 4 episodes
• Fidelity — 1× in this paper · explained in 2 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Generative Modeling — 1× in this paper · explained in 3 episodes
• Multimodal Dataset — 1× in this paper · explained in 2 episodes
• Multimodal Integration — 1× in this paper · explained in 2 episodes
• Multimodal approach — 1× in this paper · explained in 2 episodes
• Regularization — 1× in this paper · explained in 3 episodes
• Representation Learning — 1× in this paper · explained in 2 episodes
• Self-Supervised Learning — 1× in this paper · explained in 7 episodes

Sources

• LRS3-TED: a large-scale dataset for visual speech recognition
• Robust Speech Recognition via Large-Scale Weak Supervision
• MUSAN: A Music, Speech, and Noise Corpus

Related papers

• X-VC: Zero-shot Streaming Voice Conversion in Codec Space
• Autoregressive Guidance of Deep Spatially Selective Filters using Bayesian Tracking for Efficient Extraction of Moving Speakers
• Anonymization, Not Elimination: Utility-Preserved Speech Anonymization
• Towards Audio Token Compression in Large Audio Language Models
• WaveScat: Wavelet Scattering Front-Ends with Self-Supervised Features for Speech Deepfake Detection
• ProPS: Prompted Profile Synthesis for Natural Language-Conditioned Speaker Embedding Distributions