Learning to Hear Hesitation: Continual Learning for Disfluency-Aware ASR

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Learning to Hear Hesitation".

Jane: The gist This work addresses disfluent speech recognition by leveraging continual learning (CL) with explicit disfluency tokens to mitigate catastrophic forgetting and improve performance across varying disfluency distributions.

Tom: First, who's behind it and why it matters.

Title and authors: Jane: We've talked about how this paper, "Learning to Hear Hesitation: Continual Learning for Disfluency-Aware ASR," addresses the issue of information loss when ASR models are optimized to omit disfluencies by introducing explicit tokens and using continual learning to manage forgetting.

Tom: Right, and we saw that this is done across three distinct speech datasets—SME, Pitt, and Delaware—and they tested four different continual learning strategies like EWC, ER, A-GEM, and WA against them.

Lu: It’s interesting that they are using the SME Corpus from Malaysian university students to test how these models handle a specific type of language variation and disfluency distribution. That’s a very specific starting point for their continual learning setup.

Meng: So, the initial question we need to ask is what exactly is it like when a model tries to learn both general speech recognition and these new disfluency tokens simultaneously?

Jane: It's like training a student who has to learn standard English grammar while also trying to memorize every single way that someone might pause or repeat themselves. The paper seems interested in the mechanics of how that happens internally.

Tom: They looked into internal specialization using head-masking, and they found that when markers are successfully trained, a few specialized decoder cross-attention heads consistently handle the marker emission across all methods.

Lu: That suggests it’s not just one method being inherently superior; there's a shared cross-attention mechanism in the architecture that emerges once those specialized circuits are established.

Jane: So, if we look at this from a user's perspective, what does knowing there’s a shared attention pattern mean for the system I use every day?

Tom: It means that whatever learning strategy you pick will eventually lead to a similar internal structure for handling those tokens, which is good because it makes the resulting model more predictable.

Meng: But from my side, if we are deploying this at scale, we need to know if that shared mechanism is computationally efficient or if it introduces bottlenecks when processing long audio streams.

Jane: That’s a fair concern. We need to ensure that finding these specialized circuits doesn't just add complexity in terms of real-time processing costs for our production models.

Lu: The paper does explore the internal model dynamics during training, which gives us a good picture of how the model evolves from a general ASR system into one that can handle those specific speech events.

Tom: It’s like watching the model learn to recognize patterns in an audio stream; it starts by hearing words, then gradually starts paying attention to the extra stuff like fillers and disruptions.

The paper's summary: Jane: So, summarizing what we just covered, this paper is about using continual learning with explicit tokens to improve ASR for disfluent speech by showing that models can adapt across different datasets while keeping their general performance high.

Tom: It’s not just about the training; it's about the results across those three datasets—SME, Pitt, and Delaware—and how different CL methods handle these varying distributions of disfluency.

Lu: The key summary is that they are trying to bridge the gap between verbatim transcription and general-domain adaptation by using these tokens as a consistent signal during training.

Meng: So, what's the big picture takeaway for us in the AI industry? Are we just adding more labels, or is this fundamentally changing how ASR models are trained?

Jane: It suggests it’s more than just adding labels; it’s about creating a structured way to teach the model to value those extra pieces of information consistently across different speech contexts.

Tom: The paper shows that the introduction of these tokens into a pretrained backbone model creates stable token mechanisms, which is a necessary first step before any complex continual learning can happen.

Lu: And then they test how methods like A-GEM, ER, and WA perform on adapting to those different data distributions while preserving the knowledge from the original training.

Jane: It’s about showing that you can create a system that learns incrementally without forgetting its old skills as it learns new ones.

Tom: The paper demonstrates this capability by using these tokens across all three datasets, and it shows how different learning methods manage the distribution of those tokens differently during adaptation.

The paper's improvements: Jane: Now let’s talk about what the authors suggest is better than what they found, moving beyond just their initial findings in "Learning to Hear Hesitation: Continual Learning for Disfluency-Aware ASR."

Tom: They highlight that Weight Averaging achieved the strongest pWER reduction while also showing the most stable pWER performance, which they showed with their Average Incremental Word Error Rate.

Lu: That’s a strong finding because it means WA is really excellent at preserving general knowledge while adapting to new data, which is what we want for robust models.

Meng: For deployment, that stability is crucial; I’d argue that if a model keeps its base accuracy steady across different user populations, that’s more valuable than a massive jump in accuracy on one specific test set.

Jane: So it gives us a clear choice: prioritize the stability offered by WA for general robustness or look at other methods for when you need to push performance on a specific task.

Tom: On the other hand, they also found that Experience Replay showed the best marker performance with a macro-F1 score of zero point four nine during sequential adaptation, even though its pWER wasn't the absolute lowest overall <ref:2606.14391#pg1>.

Jane: So it’s not a clear winner; it really depends on whether you care more about getting high accuracy or getting high marker prediction scores.

Lu: And they also showed that ER achieved the best performance for disfluency-marker F1, showing strong generalization for markers with nearly no forgetting, as indicated by an FM score of zero point zero two during those sequential adaptation steps <ref:2606.14391#pg1>.

Meng: That suggests that if our primary goal is accurate tagging of speech events, we might want to lean towards ER because it seems to generalize the marker learning well without catastrophic forgetting on the general ASR part.

Tom: So, in short, WA is for stable baseline performance across all domains, and ER seems better when you need strong marker generalization across those challenging distributions.

Jane: The paper also points out that they found a consistent trade-off where methods that maintained the strongest pWER performance tended to under-introduce markers while those that reliably emitted markers did so at a measurable ASR cost.

Conclusion: Tom: So, wrapping up, the main thing is that CL improved sequential training for both ASR and disfluency-marker retention using explicit tokens in the paper "Learning to Hear Hesitation: Continual Learning for Disfluency-Aware ASR."

Jane: The best method really depended on your primary objective—WA showed the strongest pWER reduction while ER showed the best marker generalization and retention.

Lu: This work offers one of the first studies of continual learning specifically for modeling disfluency, giving us practical guidance for verbatim ASR adaptation and mechanistic insights into how these token behaviors are acquired in pretrained models.

Meng: For me, it’s about understanding the performance trade-offs so we can deploy models that match our specific needs to avoid wasting resources on a general solution that doesn't fit our specific requirement.

Tom: I think the consistent cross-attention pattern is a really important mechanistic finding; it explains the difference between methods like ER and WA in terms of how they handle parameter updates during adaptation.

Jane: So, we’ve seen how to use explicit tokens to make ASR models more robust against forgetting while still capturing valuable speech information.

Lu: This paper provides a solid foundation for moving forward in this area, especially regarding the structural understanding of these learned circuits.

Meng: We need to keep pushing on figuring out how to make that marker learning more efficient so it doesn't become a bottleneck at scale.