Navigating the Reality Gap: On-Device Continual Adaptation of ASR for Clinical Telephony

arXiv:2512.16401 · cs.CL · Submitted 2025-12-18 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Navigating the Reality Gap".

Tom: Automatic Speech Recognition (ASR) can significantly reduce documentation burden in clinical workflows, but standard models degrade sharply in real-world telephony settings where noisy audio, dialectal variation,

Jane: First, who's behind it and why it matters.

Title and authors: Tom: So, the paper focuses on the title "Navigating the Reality Gap: On-Device Continual Adaptation of ASR for Clinical Telephony," and it’s about showing how to adapt these models locally. They use Gram Vaani as this specific proxy for clinical speech because it represents that difficult, noisy telephony data under strict privacy rules.

Jane: It’s interesting how they frame the problem using that "reality gap" concept; it makes the challenge very concrete—it’s not just a general performance dip, but a specific mismatch between clean benchmarks and actual deployment conditions.

Lu: The authors are pushing a lot of research here by evaluating several adaptation regimes, ranging from full fine-tuning down to using parameter-efficient LoRA and streaming continual learning methods all on the device. They really explore how these different approaches perform under real constraints.

Meng: I’m curious about the specific models they test; what kind of architecture are they using as their starting point before they start applying these adaptation techniques? That foundation matters a lot for the practical feasibility we’re looking at.

Lalam: The focus on on-device constraints is huge because it directly addresses data residency concerns, which is a massive hurdle in healthcare AI deployment right now. It shows how we can build systems that respect those boundaries from the start.

The paper's summary: Tom: To summarize what they found, the core finding is that standard multilingual models like IndicWav2Vec drop from an eleven point five nine percent word error rate on clean Hindi to a much higher forty-one point seven one percent WER when tested on the Gram Vaani telephony data <ref:2512.16401#pg0>. That jump really highlights how severe the acoustic mismatch is in real-world settings.

Jane: That massive difference in performance is striking; it shows that simply having a multilingual model isn't enough if you don't account for the specific acoustic distortions inherent in phone calls, which is what this study investigates deeply.

Lu: They are using a hybrid strategy to handle this; they implement Experience Replay to ground optimization in past data and then use Elastic Weight Consolidation to stop important parameters from drifting during adaptation. That combination is the central mechanism they’re highlighting for stability.

Meng: So, instead of just throwing new audio at the model, they are actively balancing what it remembers with what it needs to learn from the new environment, which makes sense for keeping things stable on a device.

Lalam: That hybrid approach of replay and consolidation is smart because it tackles two different types of problems simultaneously: adapting to the new sound while preventing critical knowledge from being lost.

The paper's improvements: Tom: The paper suggests some really clever ways to improve this adaptation, especially concerning the interaction between Experience Replay and Elastic Weight Consolidation. They found that standard EWC with a positive lambda can actually fight against the replay-driven updates they want to use.

Jane: That’s a key finding; it means we can't just blindly apply standard stabilization techniques; we need to be more nuanced about how these two mechanisms interact when adapting on the fly.

Lu: The paper shows that relaxing that constraint by allowing a negative lambda, or Inverse EWC, actually helps reinforce those replay-driven updates instead of opposing them. It suggests EWC can function as a directional control signal under Experience Replay guidance.

Meng: If we can use the regularization strength to actively guide the learning direction rather than just passively penalizing drift, that opens up a whole new way to tune model behavior for specific acoustic conditions.

Lalam: That idea of using EWC directionally, with positive or negative lambda depending on what you want—stability versus plasticity—gives us a much finer level of control over the adaptation process itself.

Conclusion: Tom: So, to wrap up the "Navigating the Reality Gap: On-Device Continual Adaptation of ASR for Clinical Telephony" paper, they show that controlling how Experience Replay and Elastic Weight Consolidation interact is vital for robust on-device adaptation. They proved that you can use these mechanisms to guide the model intelligently through different training phases.

Jane: It really boils down to making sure the system stays stable enough not to forget what it learned from clean data, while still being plastic enough to adapt quickly to the messy, real-world audio conditions they are seeing in Gram Vaani.

Lu: The implications for future work are huge; they’re setting up a framework where we can systematically schedule stability and plasticity based on the training segment, which is a very structured way to approach continual learning problems.

Meng: Practically, this means we can design systems that manage model drift proactively rather than reacting to it after the fact during deployment in those sensitive clinical settings.

Lalam: This work gives us a concrete blueprint for building ASR systems that are not just accurate on paper but are actually resilient and safe when used where patient privacy and audio quality matter most.

Tom: Fantastic summary, team; this research really shows the path forward for making ASR tools dependable in challenging environments. We've covered a lot about managing that reality gap today.

Darshil Chauhan, Adityasinh Solanki, Vansh Patel, Kanav Kapoor, Ritvik Jain, Aditya Bansal, Pratik Narang, Dhruv Kumar

BITS Pilani, Pilani Campus, India

cs.CL

Submitted: 2025-12-18

Updated: 2026-10-02

Comments: 16 pages. Accepted at AACL-IJCNLP 2026

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 85/100

The gist: Automatic Speech Recognition (ASR) can significantly reduce documentation burden in clinical workflows, but standard models degrade sharply in real-world telephony settings where noisy audio,

Key concepts

Reality Gap
This is the difference between clean benchmark speech used for initial training and the messy, real-world audio encountered in clinical telephony. It includes issues like background noise, low-quality channels (e.g., 8 kHz), and regional dialects that standard models struggle with.
Experience Replay (ER)
ER stabilizes learning by feeding the model past data—both samples from the new target domain and general data. This keeps the model 'grounded' in previous knowledge, ensuring it doesn't forget what it learned previously while adapting to new conditions.
Elastic Weight Consolidation (EWC)
EWC stabilizes learning by identifying which model parameters are most important for old tasks and penalizing changes to those specific parameters during new training. This prevents 'catastrophic forgetting' of general language knowledge when fine-tuning for a specific local condition.

Terminology

Summary

Automatic Speech Recognition (ASR) can significantly reduce documentation burden in clinical workflows, but standard models degrade sharply in real-world telephony settings where noisy audio, dialectal variation, and strict data residency constraints prevent cloud-based adaptation. The gist: a robust multilingual model degrades from 11.59% WER on standard clean Hindi to 41.71% WER on this proxy telephony data, highlighting the critical interaction between Experience Replay (ER) and Elastic Weight Consolidation (EWC) in continual learning for on-device ASR adaptation.

The Reality Gap and Acoustic Mismatch

The paper investigates the reality gap separating clean benchmark speech from real-world telephony deployments, which is characterized by noisy audio, band-limited audio (e.g., 8 kHz), and diverse regional dialects. This mismatch is severe for models pre-trained on high-fidelity datasets. The study uses Gram Vaani as a closest available proxy for clinical telephony under strict privacy constraints, demonstrating that the IndicWav2Vec model degrades from 11.59% WER on standard clean Hindi (Kathbath) to a prohibitively high 41.71% WER on telephonic speech from Gram Vaani. This acoustic mismatch is compounded by channel attenuation and non-linear distortions introduced by low-quality telephonic channels, making traditional centralized retraining impractical due to data privacy regulations.

On-Device Continual Adaptation Regimes

The research evaluates a progression of adaptation regimes under realistic constraints, moving from full fine-tuning to parameterefficient LoRA and stream-based continual learning. The primary focus is on the interaction between two key stabilization mechanisms: multi-domain Experience Replay (ER) and Elastic Weight Consolidation (EWC). ER stabilizes learning by grounding optimization in past data, maintaining a buffer of target-domain samples from previous segments and general-domain samples. EWC stabilizes learning by penalizing drift in important parameters, using the linearized approximation of the Fisher Information to estimate parameter importance.

ER-EWC Optimization Interference and Directional Control

A central finding is that "standard EWC (λ > 0) can oppose replay-driven updates, limiting plasticity when paired with multi-domain replay. The authors demonstrate that relaxing the conventional positive sign constraint on λ reveals a more general behavior: allowing λ < 0 (Inverse EWC) reinforces replay-driven updates rather than opposing it. This suggests EWC can act as a directional control signal under ER-guided adaptation, where negative λ encourages plasticity, while scheduling allows for phase-dependent control of stability and plasticity." Specifically, the study proposes a piecewise schedule over training segments: an initial phase with λ = 0 to stabilize LoRA parameters, followed by a plasticity phase (λ 0) as adaptation saturates.

Performance Analysis Across Paradigms

The experimental design systematically compares various strategies across three datasets: Gram Vaani (Target Domain), Kathbath (Source Domain), and FLEURS (General Domain). The evaluation metrics include Word Error Rate (WER) and Character Error Rate (CER) for lexical accuracy, complemented by BERTScore to capture semantic fidelity. The progression of adaptation strategies shows that multi-domain replay provides a strong foundation for adaptation, achieving comparable final performance across different EWC configurations, indicating that multi-domain replay provides a strong foundation for adaptation. Furthermore, the analysis reveals that BERTScore trends closely follow WER across datasets, indicating consistent improvements in semantic fidelity, though neither metric is sensitive to clinically critical distinctions like negations.

Clinical Implications and Safety Considerations

The framework is explicitly designed for clinician-in-the-loop workflows where ASR acts as an assistive tool rather than an autonomous decision-maker. The design addresses privacy by ensuring raw patient audio remains local and is never transmitted to centralized servers, supporting data residency requirements. The use of EWC mechanisms helps prevent excessive forgetting of general language knowledge while adapting to local conditions, which is critical for maintaining robustness in clinical settings. Limitations noted include the reliance on WER/BERTScore as proxies for transcription quality, the need for clinician supervision in underresourced settings, and the exclusion of language models integrated into the training loop to focus on ER-EWC dynamics. The study concludes that controlling ER–EWC interaction dynamics is key to robust on-device adaptation.

How it works

  1. The system utilizes IndicWav2Vec as the acoustic backbone, a model pre-trained across 40 Indian languages, which serves as the natural starting point for studying adaptation dynamics intended for broader adoption across Indic languages.

Improvements for AI systems

As a fastidious researcher, I have analyzed the provided scientific paper, Navigating the Reality Gap: On-Device Continual Adaptation of ASR for Clinical Telephony. The core contribution is demonstrating how to effectively adapt Automatic Speech Recognition (ASR) models on resource-constrained devices (edge computing) under strict data residency constraints, specifically addressing the acoustic mismatch between clean benchmarks and noisy rural telephony.

Here are the specific improvements that can be made to AI systems based on this research:

The following improvements focus on creating robust, privacy-preserving ASR systems capable of operating in low-resource clinical environments:

  1. mathbfRobust On-Device Adaptation Framework (ER + EWC):

A system can be designed using the hybrid objective function (Algorithm 1) that combines Experience Replay (ER) with Elastic Weight Consolidation (EWC). This system will maintain a multi-domain buffer containing both target domain samples and general domain samples. During sequential data streaming, the model adapts by balancing two forces:

  • ER grounds updates in historical, relevant clinical data to prevent catastrophic forgetting of critical medical terminology.

  • EWC acts as a parameter-level constraint (regularization term) that prevents important parameters learned from previous tasks (or domains) from drifting too far during adaptation, ensuring stability.

  1. mathbfDirectional Plasticity Control via Inverse EWC:

The system can be modified to utilize the Inverse EWC formulation (V6), where the regularization strength parameter is set negatively. This transforms EWC from a purely stabilizing force into a directional control signal.

  • The improved system will actively reinforce replay-driven updates, allowing the model to explore and adapt more aggressively to new dialects or noise patterns without immediately sacrificing previously learned knowledge.
  1. mathbfTime-Dependent Stability-Plasticity Scheduling:

To prevent the instability caused by uniform negative EWC, the system should implement a scheduled control mechanism (V6.1).

  • The improved system will dynamically adjust the EWC regularization strength over training segments: starting with no regularization to stabilize initial LoRA parameters, transitioning to a plasticity phase where negative lambda encourages adaptation, and finally shifting back to positive lambda for consolidation as adaptation saturates.
  1. mathbf Parameter-Efficient Fine-Tuning (LoRA) Integration:

The system should employ Low-Rank Adaptation (LoRA) instead of full fine-tuning. This allows for rapid, parameter-efficient updates on consumer-grade edge devices while maintaining high performance gains demonstrated in the research (e.g., achieving 34% WER on the target domain).

  1. mathbf Semantic Fidelity Enhancement:

The system should incorporate a secondary metric, BERTScore, during evaluation and potentially during training via auxiliary losses (though not explicitly detailed as an integrated loss in Section 3.5). This ensures that the ASR system does not just transcribe words accurately (low WER) but also captures the correct clinical meaning (high semantic fidelity), which is crucial for safety-critical tasks like dosage transcription or negation detection.

The resulting improved AI system can perform the following specific functions:

  1. mathbfClinical Transcription in Low-Resource Settings:

The system can accurately transcribe spoken patient information from rural healthcare helplines, even with severe acoustic challenges (8kHz band-limited audio, high noise).

  1. mathbfOn-Device Deployment with Privacy Assurance:

Because adaptation is localized and continual, the system can run entirely on a local device (e.g., a hospital tablet), ensuring that raw patient audio data never leaves the secure environment, satisfying strict data residency regulations.

  1. mathbfContinuous Domain Adaptation:

The system can continuously learn and adapt to evolving regional dialects or changes in clinic protocols over time, without suffering from catastrophic forgetting of general linguistic knowledge (Kathbath/FLEURS domains).

  1. Anticipatory Model Drift Management:

By using the scheduled EWC mechanism, the system can be designed to maintain a safe level of stability for critical medical concepts while allowing necessary plasticity for new acoustic conditions or language variations encountered in real-time.

Sources

Related papers