When Audio-Language Models Fail to Leverage Multimodal Context for Dysarthric Speech Recognition

arXiv:2605.02782 · cs.AI, cs.CL, eess.AS · Submitted 2026-08-18 · Read on arXiv

Pehuén Moure, Niclas Pokel, Bilal Bounajma, Yingqiang Gao, Roman Boehringer, Longbiao Cheng, Shih-Chii Liu

cs.AI, cs.CL, eess.AS

Submitted: 2026-08-18

Updated: 2026-08-20

Code: https://github.com/fixie-ai/ultravox

Terminology

• Fine-Tuning — 25× in this paper · explained in 5 episodes
• Hallucination — 13× in this paper · explained in 3 episodes
• Adaptation — 9× in this paper · explained in 1 episode
• Frozen model — 8× in this paper · explained in 2 episodes
• In-Context Learning — 7× in this paper · explained in 4 episodes
• Benchmark — 6× in this paper · explained in 2 episodes
• Speech Accessibility Project — 6× in this paper
• Failure Modes — 5× in this paper · explained in 3 episodes
• Ablation — 3× in this paper · explained in 1 episode
• Data Augmentation — 3× in this paper · explained in 7 episodes
• LoRA Adaptation — 3× in this paper · explained in 2 episodes
• Semantic Similarity — 3× in this paper · explained in 2 episodes
• Accuracy — 2× in this paper · explained in 1 episode
• Automatic Speech Recognition (ASR) — 2× in this paper · explained in 2 episodes
• Generalization — 2× in this paper · explained in 13 episodes
• Personalization — 2× in this paper · explained in 2 episodes
• Chain-of-Thought (CoT) — 1× in this paper · explained in 4 episodes
• Commentary — 1× in this paper · explained in 2 episodes
• Core — 1× in this paper · explained in 2 episodes
• Distribution Shift — 1× in this paper · explained in 5 episodes
• Domain Shift — 1× in this paper · explained in 4 episodes
• Error Decomposition — 1× in this paper · explained in 3 episodes
• Self-Supervised Learning — 1× in this paper · explained in 7 episodes
• Structural Constraints — 1× in this paper · explained in 2 episodes

Sources

• Zero-Shot Recognition of Dysarthric Speech Using Commercial Automatic Speech Recognition and Multimodal Large Language Models
• Qwen2-Audio Technical Report
• A Survey on In-context Learning
• Gemini: A Family of Highly Capable Multimodal Models
• Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models
• Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training
• LoRA: Low-Rank Adaptation of Large Language Models
• Voxtral
• Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs
• Adversarial NLI: A New Benchmark for Natural Language Understanding
• GPT-4o System Card
• Aligning ASR Evaluation with Human and LLM Judgments: Intelligibility Metrics Using Phonetic, Semantic, and NLI Approaches
• Adapting Foundation Speech Recognition Models to Impaired Speech: A Semantic Re-chaining Approach for Personalization of German Speech
• Robust Speech Recognition via Large-Scale Weak Supervision
• Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition
• Qwen3-ASR Technical Report
• Personalized Fine-Tuning with Controllable Synthetic Speech from LLM-Generated Transcripts for Dysarthric Speech Recognition
• Qwen3-Omni Technical Report
• MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe
• BERTScore: Evaluating Text Generation with BERT

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
• VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection