LLM-HB: Language-Aware LLM-Guided Hotword Biasing for Code-Switching ASR

summary

The gist

The gist The framework incorporates a mixture-of-experts (MoE) adaptor and an auxiliary language prediction head to better capture and differentiate multilingual representations, while hotwords are

In short

The LLM-HB framework enhances Automatic Speech Recognition (ASR) for code-switching speech by combining a Mixture-of-Experts (MoE) adaptor, an auxiliary language prediction head, and LLM-based hotword biasing. This approach dynamically specializes speech representations for different languages and uses targeted prompting to improve recognition accuracy in multilingual scenarios.

Key concepts

Code-Switching (CS) Speech: This refers to speech where a speaker alternates between two or more languages within a single continuous utterance or conversation. It presents a major challenge for ASR systems because the rapid language shifts confuse models trained primarily on monolingual data, making accurate transcription difficult.
Mixture-of-Experts (MoE) Adaptor: The MoE adaptor dynamically selects and combines different neural network 'experts' to process speech features. In this framework, it specializes the speech embeddings for different languages, allowing the model to create language-specific representations that better capture the nuances of each language being spoken.
LLM-Guided Hotword Biasing: This technique uses a frozen Large Language Model (LLM) to generate special prompts that guide the ASR system toward recognizing specific 'hotwords.' By incorporating these LLM-derived signals into the training loss, the model learns to prioritize and improve recognition for these key phrases in code-switched input.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 20 episodes
Retrieval-Augmented Generation (RAG) — explained in 18 episodes
Federated Learning — explained in 17 episodes
Quantum Fisher Information (QFI) — explained in 16 episodes
Flow Matching — explained in 15 episodes

This episode discusses

The paper

LLM-HB: Language-Aware LLM-Guided Hotword Biasing for Code-Switching ASR · Read on arXiv

Yuxuan He, Genshun Wan, Pengcheng Li, Gongping Huang, Jianqing Gao

Auditory Cognition and Computational Acoustics Lab, Shanghai Jiao Tong University · AI research Institute, iFLYTEK Company Limited · School of Electronic Information, Wuhan University

DOI: 10.21437/Interspeech.2026-1113

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "LLM-HB: Language-Aware LLM-Guided Hotword Biasing for Code-Switching ASR".

Jane: The gist The framework incorporates a mixture-of-experts (MoE) adaptor and an auxiliary language prediction head to better capture and differentiate multilingual representations, while hotwords are injected through LLM-based prompting.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: We're wrapping up this look at LLM-HB, which stands for Language-Aware LLM-Guided Hotword Biasing for Code-Switching ASR.

Jane: It sounds like a lot of technical terms there, Tom. Can you give us the plain story on what this whole paper is actually about?

Lu: They are using a mixture of experts adaptor to dynamically integrate speech and text representations, plus an auxiliary language head for language identity supervision, and they inject hotwords through LLM prompting.

Meng: So it’s not just trying to recognize words; it’s trying to figure out which language is being spoken at any given moment, and using that knowledge to boost the recognition.

Tom: Exactly. And they inject hotwords—those specific phrases—into the model using LLM prompting, which acts like a strong nudge for the ASR system when it hears those key words.

Jane: So instead of just listening to audio and guessing what’s happening, this system is actively guided by language context and those hotword signals. How does that actually change things for someone who just wants a better tool?

Lu: The main result they show on the ASRU2019-CS dataset is a MER score of five point eight five percent when they use all three parts together, which was a big improvement over the baseline of seven point three four percent.

Meng: I gotta ask, that’s good for the numbers, but does that work on my actual customer service lines? Because I’m not running this on some clean benchmark dataset.

Tom: Well, they did test it on the codeswitching data and the hotword performance improved really significantly—the B-MER dropped from twenty-six point nine one percent all the way down to eight point nine nine percent. That suggests a real practical benefit for those specific scenarios.

Jane: So, even with limited training data, this framework can pull in a lot of accuracy by using that language awareness and hotword biasing. What does that imply for the future of multilingual speech recognition?

Lu: It shows that dynamically specializing the representations with the mixture of experts is a way to handle language boundaries better than just one fixed model.

Meng: For deployment, it means we have a clearer path on how to inject domain-specific information like hotwords without completely retraining the whole thing from scratch.

Jane: Next time we talk about these kinds of complex systems, we should look at how they handle things that sound almost the same but mean different things.

Conclusion: Tom: So we’re wrapping up this look at LLM-HB, which stands for Language-Aware LLM-Guided Hotword Biasing for Code-Switching ASR.

Jane: It sounds like a lot of technical terms there, Tom. Can you give us the plain story on what this whole paper is actually about?

Meng: So it’s not just trying to recognize words; it’s trying to figure out *which* language is being spoken at any given moment, and using that knowledge to boost the recognition.

Tom: Exactly. And they inject hotwords—those specific phrases—into the model using LLM prompting, which acts like a strong nudge for the ASR system when it hears those key words.

Jane: So, instead of just listening to audio and guessing what’s happening, this system is actively guided by language context and those hotword signals. How does that actually change things for someone who just wants a better tool?

Lu: The paper shows that dynamically specializing the representations with the mixture of experts is a way to handle language boundaries better than just one fixed model.

Meng: For deployment, it means we have a clearer path on how to inject domain-specific information like hotwords without completely retraining the whole thing from scratch.

Tom: This whole paper, LLM-HB, basically shows that you can use large language models not just for generating text, but to actively guide speech recognition in these tricky code-switching situations by adding those hotword prompts.

Jane: It sounds like a very structured way of combining language discrimination with targeted word emphasis. So what does this mean for how we approach similar problems in other areas?

More episodes

← Home