HALAS: A Human-Annotated Dataset of Hallucinations of Modern ASR Systems

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "HALAS: A Human-Annotated Dataset of Hallucinations of Modern ASR Systems".

Jane: Here's the gist of the paper. The gist The HALAS dataset establishes the first rigorous non-artificial benchmark for the detection and mitigation of ASR hallucinations by providing human-annotated spans from seven state-of-the-art ASR models on real earnings call recordings.

Tom: Dataset Creation Methodology The researchers introduced HALAS, which is described as "the first publicly available dataset of human-labeled hallucinations from seven state-of-the-art ASR models on recordings from the Earnings 22 dataset" (Page 1).

Jane: The dataset was created using the Earnings 22 (E22) dataset

1: which "contains earnings calls in English conducted by speakers from 27 countries" (Page 5).

Tom: They utilized "7 state-of-the-art ASR models that achieve low WER results across various datasets in the Open ASR Leaderboard

13: " (Page 2).

Title and authors: Tom: The title itself tells you a lot about what they're doing, focusing on human-annotated hallucinations and modern ASR systems. It sets the expectation that this isn't some synthetic test set.

Jane: So what they really found is that all seven of these state-of-the-art models are prone to making these errors when exposed to real audio, and crucially, the errors they make actually share vocabulary patterns across all the different systems they tested.

Lu: That cross-model vocabulary overlap is significant because it suggests that the underlying issue isn't unique to one specific model architecture; it points toward more general weaknesses in how ASR systems handle natural language.

Meng: I mean, so if we only looked at one system and saw a lot of errors, we might think that specific model is broken, but this shows they are all hitting similar linguistic weak spots on real speech.

Tom: Exactly. And they also found something interesting about the quality of the speech—hallucinations still happen even for audio that has a low Word Error Rate, which means it sounds almost perfectly transcribed.

The paper's summary: Jane: Beyond just having this dataset, what are the actual applications or suggested improvements they propose for using HALAS? What's the next step here?

Tom: They position HALAS as a benchmark, which means it’s designed to let people evaluate different detection methods. They set up two main approaches for this evaluation: using proxy metrics and testing state-of-the-art detection methods directly.

Lu: When they used proxy metrics like CER and SeMaScore, those actually performed slightly better than the other metrics they tested, which suggests that combining those categories of measurement might lead to a better detection score.

Meng: That makes sense for initial screening tools because you want a fast way to filter out noise before you spend time training heavy classifiers on complex inputs. What about the actual classifier performance?

Tom: They trained an XGBoost classifier using all seven text proxy metrics across different conditions, like OWN, OTHER, and ALL regimes. This setup achieved a mean ROC-AUC score of zero point eight two nine in OWN, zero point eight two eight in OTHER, and zero point eight three five in ALL conditions <ref:2606.23048#pg3>.

Jane: And they also showed performance on top-tier models like Whisper large v3 where a multi-layer detector worked better than the single layer baseline and hit an F1 score of zero point five six one percent <ref:2606.23048#pg2>.

The paper's improvements: Tom: So to summarize, the main contribution of HALAS is providing this rigorously created dataset that confirms all seven models hallucinate with errors showing strong cross-model similarities and distributions skewed toward small sets of phrases.

Jane: The real utility they demonstrate is using HALAS to benchmark detection methods against these naturally occurring, human-annotated hallucinations, which I think is a substantial step forward for how we evaluate these systems in the real world.

Lu: It really solidifies the idea that these models aren't just making random mistakes; they have predictable patterns in their errors that are shared across different systems, which gives us a better direction for fixing them.

Meng: From an engineering standpoint, having this benchmark means we can finally test if our detection systems actually generalize across different ASR architectures, which is exactly what we need to worry about when deploying something at scale.

Lalam: I’m confident that the findings regarding the cross-model vocabulary overlap in HALAS provide a very reliable measure for understanding where these errors originate in modern ASR systems, and I am quite sure of that conclusion.

Conclusion: Tom: So, wrapping up this discussion on HALAS: A Human-Annotated Dataset of Hallucinations of Modern ASR Systems, we've seen how all seven state-of-the-art models hallucinate on real earnings call recordings and how we can finally benchmark detection methods against that data.

Jane: We confirmed that the hallucination rate varies between twenty-one point four percent and forty-three point eight percent across those models, and they also showed looping only occurred in one point one percent of cases on average for the entire set <ref:2606.23048#pg3,hallucination rate varies between 21.4>.

Lu: That variation confirms every single model hallucinates on this HALAS audio data, which is significant because it shows it’s a widespread problem across the board, not just an issue with one specific model architecture.

Meng: So not just one model failing, but they are all prone to hallucinations even when the Word Error Rate is low, and that suggests a fundamental issue with how these models process natural speech at this level.

Tom: And they also looked at looping, and that only occurred in one point one percent of cases on average for the entire set.

Jane: It’s interesting because they also found that the distribution of the hallucinated phrases was heavily skewed toward a small set of the most frequent phrases for each model.

Lu: They quantified this by showing that on average, fifty-five percent of all marked hallucinations corresponded to just the top ten phrases for any given model, which is a very specific pattern.

Meng: That tells you something about how predictable these errors are; they aren't random nonsense, they cluster around certain common utterances that we can actually try to target.

Tom: So to summarize, the authors introduced HALAS as a novel dataset for studying ASR hallucinations on real speech recordings, confirming that all seven models hallucinate with errors showing strong cross-model similarities and distributions skewed toward small sets of phrases.

Jane: The study successfully demonstrated the utility of HALAS by benchmarking detection methods against naturally occurring, human-annotated hallucinations, which is a big step for evaluation in this area.

Lu: It solidifies the idea that these models aren't just making random mistakes; they have predictable patterns in their errors that are shared across different systems.

Meng: From an engineering standpoint, having this benchmark means we can finally test if our detection systems actually generalize across different ASR architectures, which is what we need to worry about at scale.

Lalam: I’m confident that the findings regarding the cross-model vocabulary overlap in HALAS provide a very reliable measure for understanding where these errors originate in modern ASR systems.

Tom: So, with this work on HALAS: A Human-Annotated Dataset of Hallucinations of Modern ASR Systems, we have a much clearer picture of the real challenges facing speech transcription systems. Next up, we’re looking at how we can actually use these detection methods to fix them or at least filter out that noise in production environments.