Text-only adaptation in LLM-based ASR through text denoising
cs.SD, cs.CL, cs.LG, eess.AS
Submitted: 2026-01-28
Updated: 2026-09-22
Comments: Notice: this version has been superseded by a revised version published at Interspeech: https://www.isca-archive.org/interspeech_2026/burdisso26_interspeech.html
Code: https://github.com/skit-ai/SpeechLLM
License: http://creativecommons.org/licenses/by/4.0/
The gist: Adapting large language model (LLM)-based automatic speech recognition (ASR) systems to new domains using text-only data is a significant yet underexplored challenge.
Terminology
Abstract
Adapting large language model (LLM)-based automatic speech recognition (ASR) systems to new domains using text-only data is a significant yet underexplored challenge. Standard fine-tuning of the LLM on the target domain text often disrupts the critical alignment between the speech and text modality learned by the projector, degrading performance. We introduce a novel text-only adaptation method that frames this process as a text denoising task. Our approach trains the LLM to recover clean transcripts from noisy inputs. This process effectively adapts the model to a target domain while preserving cross-modal alignment. Our solution is lightweight, requiring no architectural changes or additional parameters. Extensive evaluation on two datasets demonstrates up to 22.1% relative improvement, outperforming recent state-of-the-art text-only adaptation methods.
Sources
- Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models
- Qwen2-Audio Technical Report
- An Embarrassingly Simple Approach for LLM with Strong ASR Capacity
- Can Generative Large Language Models Perform ASR Error Correction?
- SpeechVerse: A Large-scale Generalizable Audio Language Model
- Approaching Dialogue State Tracking via Aligning Speech Encoders and LLMs
Related papers
- Few-Shot Open-Set Audio Classification via Transductive Prototype Refinement and Class Logit Enhancement
- Spectral Masking and Interpolation Attack (SMIA): A Black-box Adversarial Attack against Voice Authentication and Anti-Spoofing Systems
- AVMeme Exam: A Multimodal Multilingual Multicultural Benchmark for LLMs' Contextual and Cultural Knowledge and Thinking
- SoundWeaver: Compositional Warm-Starting for Text-to-Audio Diffusion Serving
- WASIL: In-the-Wild Arabic Spoken Interactions with LLMs
- Efficient Audiovisual Speech Processing via MUTUD: Multimodal Training and Unimodal Deployment