Whisper-Aware LLM: Self-Supervised Uncertainty Learning for Robust Whispered Speech Recognition

The paper introduces the Whisper-Aware LLM, a framework designed to address the challenges of whispered speech recognition in ASR systems. The core problem is that whispered speech, characterized by the absence of fundamental frequency (F0) and harmonic structure, creates signal ambiguity that leads to two failure modes: failing to capture whispered speech or hallucinatory transcription of noise. The authors state: The signal ambiguity of whispered speech drives ASR systems toward two opposing failure modes: failing to capture whispered speech or hallucinatory transcription of noise.

The framework enhances a standard Audio-LLM by integrating a lightweight Uncertainty Perception Module (UPM) and a Confidence-Fused Decoding mechanism. The UPM is trained using two self-supervised tasks: F0 contour prediction and masked spectrum reconstruction. The F0 contour prediction task is motivated by the finding that the lack of glottal information is the single most critical factor degrading recognition performance. The masked spectrum reconstruction task captures the unstructured, noise-like nature of the whisper spectrum.

The Confidence-Fused Decoding mechanism has two components: a global instruction embedding and attention modulation. The global uncertainty vector is transformed by an MLP into an instruction embedding that acts as a high-level system prompt. The attention modulation injects learned confidence scores as additive bias to attention scores for acoustic frames, allowing the model to dynamically pay less attention to acoustic frames deemed unreliable by the UPM.

The training uses a three-stage strategy: Stage 1 pre-trains the UPM exclusively on self-supervised objectives while the backbone remains frozen; Stage 2 trains the audio encoder, adapter, UPM, and decoding interfaces while keeping the LLM decoder frozen; Stage 3 fine-tunes the entire model end-to-end with LoRA applied to the LLM decoder.

Experimental results show the model achieves state-of-the-art performance on whispered speech. On the AISHELL6-Whisper test set, it achieves a CER of 1.31% on whispered speech, representing a relative CER reduction of 17% over the previous best compared to Seed-ASR's 1.58%. The model also achieves the lowest error rate on normal speech (0.63%), indicating specialization for whispers does not compromise general ASR capabilities.

On English benchmarks, the model consistently outperforms all baselines across normal and whispered conditions for US and Singaporean accents. For example, on whispered US speech, it achieves 5.92% WER compared to the best baseline of 6.35%, and on whispered Singaporean speech, it achieves 10.81% WER compared to 11.63% for the best baseline.

For general ASR performance, the model achieves a CER of 1.34% on AISHELL-1 and a WER of 1.91% on LibriSpeech-clean, placing it on par with other top-tier systems.

The reliability evaluation on the Noise Hallucination Set shows the model achieves a Hallucination Rate of only 4.5%, compared to over 25% for strong baselines like S7 and S5, with the highest rate reaching 29.3% for S6. The authors note: This remarkable improvement demonstrates that by learning to perceive signal uncertainty, our model effectively learns when not to speak, drastically mitigating the risk of generating spurious text from non-speech audio.

The ablation study shows that using only frame-wise Attention Modulation improves CER from 3.98% to 3.45%, the Global Instruction alone reduces CER to 1.84%, and the full model achieves the best performance at 1.31%. The authors conclude: The Global Instruction, however, proves to be the primary driver of performance, slashing the CER to 1.84% by providing high-level, strategic awareness of signal quality.

The paper's contributions are summarized as: a novel framework integrating self-supervised uncertainty perception into an Audio-LLM, two physics-informed self-supervised tasks for label-free uncertainty learning, a Confidence-Fused Decoding mechanism for dynamic generative control, and a systematic three-stage training protocol for robust representation learning.

Improvements to AI Systems:

1. Integrate a lightweight Uncertainty Perception Module (UPM) into any audio-based LLM to explicitly model signal-level uncertainty (e.g., missing F0, noise-like spectra) before decoding. This module can be trained with two self-supervised tasks—F0 contour prediction and masked spectrum reconstruction—requiring no manual labels, making it adaptable to other low-resource or degraded audio domains (e.g., whispered speech, heavy background noise, or telephonic audio).

2. Add a Confidence-Fused Decoding mechanism with two components:

• Global Instruction Embedding: Transform the UPM's global uncertainty vector via an MLP into a system-level prompt that tells the LLM whether the input is reliable, partially reliable, or unreliable. This enables the model to adjust its overall behavior (e.g., be conservative or transcribe only if confident) without retraining the core decoder.

• Attention Modulation: Inject learned confidence scores as additive bias to attention weights for each acoustic frame. This allows the LLM to dynamically down-weight unreliable frames during generation, reducing hallucination from noise and improving focus on genuinely informative segments.

3. Implement a three-stage training protocol that separates representation learning from decoder fine-tuning:

• Stage 1: Pre-train the UPM alone on self-supervised objectives (frozen backbone) to learn robust uncertainty features.

• Stage 2: Train the audio encoder, adapter, UPM, and decoding interfaces while keeping the LLM frozen—ensuring the model learns to map uncertainty to control signals without corrupting language knowledge.

• Stage 3: Fine-tune the entire model end-to-end with LoRA on the LLM decoder, allowing task-specific adaptation with minimal catastrophic forgetting.

4. Use physics-informed self-supervised tasks (F0 prediction and masked spectrum reconstruction) to make the UPM sensitive to the root cause of whisper degradation—absence of glottal excitation—rather than relying on generic acoustic features. This improves robustness to unseen whisper conditions and accents.

What the Improved AI System Can Do:

• Transcribe whispered speech with high accuracy (e.g., 1.31% CER on AISHELL6-Whisper, 17% relative improvement over prior SOTA) while maintaining or improving performance on normal speech (0.63% CER on AISHELL6-Normal, 1.91% WER on LibriSpeech-clean).

• Dynamically suppress hallucination on non-speech or noise-only inputs, reducing hallucination rate from >25% (baselines) to 4.5% on a Noise Hallucination Set—effectively knowing when not to speak.

• Generalize across accents and languages (e.g., US and Singaporean English, Mandarin) without task-specific retraining, thanks to the label-free UPM and confidence-based decoding.

• Operate in low-resource or degraded audio environments (e.g., far-field, telephony, or whispered speech) by learning to perceive uncertainty rather than overfitting to clean speech patterns.

• Provide interpretable confidence signals at both global (utterance-level) and local (frame-level) scales, enabling downstream systems (e.g., hearing aids, voice assistants) to make safer decisions when audio quality is poor.

The signal ambiguity of whispered speech drives ASR systems toward two opposing failure modes: failing to capture whispered speech or hallucinatory transcription of noise. This paper introduces the Whisper-Aware LLM, a framework that teaches an Audio-LLM to perceive and react to this uncertainty. Our model develops an intrinsic self-awareness by learning to quantify the physical deficiencies of acoustic signals through targeted self-supervised tasks. This learned uncertainty is then operationalized via a novel Confidence-Fused Decoding mechanism, which provides both high-level instructions and frame-level attention modulation to the LLM decoder. Our experiments confirm the effectiveness of this approach. The model sets a new state-of-the-art on whispered speech with a 17% relative CER reduction on AISHELL6-Whisper. At the same time, it directly addresses the reliability trade-off, with hallucination rates dropping from over 25% to 4.5%.