Diffusion Large Language Models for Visual Speech Recognition

arXiv:2605.28456 · cs.AI, cs.CV, eess.AS · Submitted 2026-05-27 · Read on arXiv

cs.AI, cs.CV, eess.AS

Submitted: 2026-05-27

Updated: 2026-09-01

Comments: Accepted to EMNLP 2026. Code: https://github.com/JeongHun0716/dllm-vsr

Code: https://github.com/JeongHun0716/dllm-vsr

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Transformer — 17× in this paper · explained in 2 episodes
• Large Language Model — 12× in this paper
• Decode — 8× in this paper · explained in 2 episodes
• Diffusion Large Language Model — 7× in this paper
• Alignment — 6× in this paper · explained in 1 episode
• Two-stage training — 4× in this paper · explained in 2 episodes
• Accuracy — 3× in this paper · explained in 1 episode
• Adaptation — 3× in this paper · explained in 1 episode
• Benchmark — 3× in this paper · explained in 2 episodes
• Generalization — 3× in this paper · explained in 13 episodes
• Diffusion Model — 2× in this paper · explained in 8 episodes
• Diffusion Models — 2× in this paper · explained in 6 episodes
• Existing Visual Speech Recognition — 2× in this paper
• Instruction Tuning — 2× in this paper · explained in 2 episodes
• Ablation — 1× in this paper · explained in 1 episode
• Ablation study — 1× in this paper · explained in 1 episode
• Block Diffusion — 1× in this paper · explained in 2 episodes
• Computational overhead — 1× in this paper · explained in 2 episodes
• Connectionist Temporal Classification — 1× in this paper
• Iterative Refinement — 1× in this paper · explained in 3 episodes
• KV cache — 1× in this paper · explained in 3 episodes
• Large Language Model (LLM) — 1× in this paper · explained in 3 episodes
• Regularization — 1× in this paper · explained in 3 episodes
• Representation Learning — 1× in this paper · explained in 2 episodes
• Robustness — 1× in this paper · explained in 5 episodes
• Sample Efficiency — 1× in this paper · explained in 4 episodes
• Self-Supervised Learning — 1× in this paper · explained in 7 episodes
• Tokenization — 1× in this paper · explained in 2 episodes
• Transformer Architecture — 1× in this paper · explained in 2 episodes

Sources

• LRS3-TED: a large-scale dataset for visual speech recognition
• LipNet: End-to-End Sentence-level Lipreading
• Improving Variable-Length Generation in Diffusion Language Models via Length Regularization
• The Llama 3 Herd of Models
• Transformer-Based Video Front-Ends for Audio-Visual Speech Recognition for Single and Multi-Person Video
• Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
• Dream-Coder 7B: An Open Diffusion Language Model for Code
• Qwen2.5 Technical Report
• Diffusion LLM with Native Variable Generation Lengths: Let [EOS] Lead the Way
• Dream 7B: Diffusion Large Language Models
• LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning
• Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decoding

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
• VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection