Pretrained self-supervised speech models can recognize unseen consonants

arXiv:2606.11542 · cs.CL, cs.AI · Submitted 2026-06-10 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Pretrained self-supervised speech models can recognize unseen consonants".

Jane: The gist Pretrained self-supervised speech models can recognize click consonants as accurately as other speech sounds, suggesting that self-supervision enables generalization across human speech sounds including rare phonemes.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Now let's look at what they actually found in their results. The main finding is pretty direct: clicks are recognized more accurately than non-click sounds in both languages tested, Gui and West!Xoon.

Jane: They used a Wilcoxon signed-rank test to compare the error rates of click consonants against the error rates of non-click phonemes under greedy decoding, and it showed that clicks had significantly lower error rates with a W score of zero and a p value of zero point zero one six.

Lu: That statistical significance is key because it points to a measurable advantage for these rare sounds in this context, which supports the idea that self-supervised pretraining can handle complexity.

Meng: They also looked at vowels, and they found that vowels actually showed relatively higher error rates in both languages because the acoustic differences between vowel qualities can be more gradual than those for clicks.

Tom: So, to summarize what they found, the study concluded that these pretrained models consistently gave lower phoneme error rates for click consonants compared to non-click ones.

Jane: This suggests that the way these self-supervised speech models are trained gives them a strong ability to adapt to speech sounds they haven't encountered before in their initial training data.

The paper's summary: Tom: So, what about the implications? The paper points out that this success suggests the self-supervised training paradigm of these models is quite adaptable when faced with speech sounds that weren't in the original pretraining set.

Jane: It means these models aren't just good at repeating what they've heard; they have a structural capability to generalize better to things outside their initial data distribution.

Lu: The improvement here is showing that this specific self-supervised approach doesn't get completely stuck when the input sounds are typologically rare, like clicks from Khoisan languages.

Meng: From an engineering standpoint, this is promising because it means we don't necessarily need to gather massive, perfectly balanced datasets for every single sound type just to make a model robust.

Tom: It also implies that the pretraining process itself might be teaching the models some underlying features about speech that are useful even for very uncommon sounds.

Jane: It moves the conversation away from just worrying about data scarcity and toward understanding what's actually learned during that self-supervised pretraining phase.

The paper's improvements: Tom: So, to wrap up this discussion on "Pretrained self-supervised speech models can recognize unseen consonants," the main conclusion is that these models consistently achieved lower phoneme error rates for click consonants than for non-click phonemes in both Gui and West!Xoon.

Jane: This finding strongly suggests that the self-supervised training approach exhibits a strong adaptability to speech sounds that were not present in the initial pretraining data.

Lu: It really shows that these models can perform better when they encounter sounds outside of the dominant language distributions they were originally exposed to, which is a big thing for multilingual systems.

Meng: For practical application, this means we can expect these models to handle more diverse and less common speech patterns without needing an entirely new massive dataset just for those specific rare sounds.

Lalam: I think what this means for the culture is that it opens up the possibility of creating speech recognition tools that serve a wider variety of human languages more fairly.

Tom: Right, so we've looked at how these models handle clicks from Gui and West!Xoon, and the results show they recognize those clicks better than other sounds. It’s a solid piece of evidence for the adaptability of self-supervised learning in this area.

Conclusion: Tom: So we’ve looked at how pretrained self-supervised speech models handle clicks in Gui and West!Xoon, and the results show they recognize those clicks more accurately than non-click sounds in both languages.

Jane: That's right, Tom, it really shows that these models can adapt to sounds they weren't trained on before without needing a whole new dataset just for those rare consonants.

Lu: It’s interesting because this suggests the self-supervised training method has a real capacity for generalization beyond the languages it sees during pretraining.

Meng: From an engineering angle, it means we don't always need perfectly balanced data to get good performance on unusual phonemes in our ASR systems.

Lalam: I think this capability is huge because it means future language tools can start recognizing sounds from places we haven't even heard before, which really expands how we can process human speech.

Tom: Exactly, Lalam, so the paper "Pretrained self-supervised speech models can recognize unseen consonants" gives us a clear picture of this adaptability.

Jane: It’s a solid piece of work because it proves that the way these models are trained supports generalization to typologically rare sounds.

Lu: I think it really challenges our old assumptions about how much data we need to collect for every single sound type in speech recognition.

Meng: The main thing is seeing that statistical difference—that W score and p value—it’s concrete evidence that the performance improvement isn't just noise.

Lalam: It changes how we think about building future language models, showing them a more robust foundation from the start.

Tom: Well, this was a great look at clicks, but where do we go from here for these models?

University of Notre Dame, USA · University at Buffalo, USA

cs.CL, cs.AI

Submitted: 2026-06-10

Updated: 2026-10-08

Comments: 6 pages, 3 figures, 3 tables, presented at Interspeech 2026

Code: https://github.com/kensho-technologies/pyctcdecode

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 82/100

The gist: The gist Pretrained self-supervised speech models can recognize click consonants as accurately as other speech sounds, suggesting that self-supervision enables generalization across human speech

Key concepts

Self-Supervised Speech Models: These are advanced artificial intelligence models trained on massive amounts of raw audio data without needing human-labeled transcripts during the initial training. They learn deep, contextual representations of speech by predicting parts of the audio, allowing them to understand the underlying structure and nuances of human language.
Click Consonants: These are speech sounds produced by creating a sudden release of air through a constricted space in the mouth, often involving clicking or popping noises. They are rare in many languages, especially those from the Khoisan family, making them difficult for standard speech recognition systems to identify.
Generalization: This refers to a model's ability to perform well on data it was not explicitly trained on. The study investigates whether self-supervised models can generalize their learning from common speech sounds (like those in high-resource languages) to recognize rare, typologically uncommon sounds like clicks.

Terminology

Generalization — 7× in this paper · explained in 13 episodes
Adaptability — 6× in this paper · explained in 1 episode
Fine-Tuning — 6× in this paper · explained in 5 episodes
Generative AI — 5× in this paper · explained in 2 episodes
Phoneme Error Rate — 5× in this paper
Low-Resource Languages — 3× in this paper · explained in 4 episodes
Representation Learning — 3× in this paper · explained in 2 episodes
Transformer — 3× in this paper · explained in 2 episodes
Connectionist Temporal Classification — 2× in this paper
Self-Supervised Learning — 2× in this paper · explained in 8 episodes
Accuracy — 1× in this paper · explained in 1 episode
Adaptation — 1× in this paper · explained in 1 episode
Alignment — 1× in this paper · explained in 1 episode
Automatic Speech Recognition (ASR) — 1× in this paper · explained in 2 episodes
Benchmark — 1× in this paper · explained in 2 episodes
Central Kalahari Game Reserve — 1× in this paper
Character Error Rate — 1× in this paper
Clustering — 1× in this paper · explained in 2 episodes
Cross-lingual Transfer — 1× in this paper · explained in 2 episodes
Initial Data — 1× in this paper · explained in 2 episodes
Regularization — 1× in this paper · explained in 3 episodes
Robustness — 1× in this paper · explained in 5 episodes
Transferability — 1× in this paper · explained in 4 episodes

Summary

The gist

Pretrained self-supervised speech models can recognize click consonants as accurately as other speech sounds, suggesting that self-supervision enables generalization across human speech sounds including rare phonemes.

Introduction and Motivation

Modern pretrained self-supervised automatic speech recognition models are trained on large-scale audio data to encode speech into contextualized representations. However, their training data are heavily skewed toward high-resource languages with little data from low-resource languages, raising concerns about the potential underrepresentation of typologically uncommon speech sounds such as click consonants primarily found in Khoisan languages. This leads to the central research question: Can these models recognize click consonants as accurately as other speech sounds. The authors investigate whether pretrained selfsupervised multilingual speech models can accurately recognize click consonants by fine-tuning several widely used selfsupervised architectures on data from Gui and West!Xoon.

Methodology

The study fine-tuned several pretrained selfsupervised speech models, specifically Wav2Vec 2.0 and HuBERT, on data from two click-rich Khoisan languages: Gui and West!Xoon. The researchers compared recognition performance between click and non-click phonemes to assess whether typologically uncommon sounds are disadvantaged in these models.

The following models were compared in the experiments:

wav2vec2-large-xlsr-53, wav2vec2-xls-r-300m, and mms-1b all exhibited similar error rates in Gui, as Figure 1b shows.

The fine-tuning process involved setting specific hyperparameters:

The model training employs the same hyperparameters.
Each training is run for 10 epochs with a learning rate of 0.0003 and a batch size of 8, optimized by the AdamW.
The first 100 steps are reserved as warm-up steps.

Data Construction

The datasets consist of two Khoisan languages, Gui and West!Xoon, which belong to genealogically separate language families, Khoe–Kwadi and Tuu, respectively.

(a) Gui dataset:

Total length (s) 18616 2044 20660.

(b) West!Xoon dataset:

Total length (s) 5006 1410 6416.

The phonological inventory of Gui comprises approximately 90 phonemes, including 52 click consonants and 38 non-click consonants. The West!Xoon language has a phonological inventory with 43 click consonants, making it the most consonant-rich language in the Tuu language family.

Experimental Results

The results demonstrated that clicks are recognized more accurately than non-click phonemes in both languages. Specifically, a Wilcoxon signed-rank test comparing error rates of click consonants and non-click phonemes under greedy decoding showed that click consonants are recognized with significantly lower error rates (W = 0, p = 0.016). This observation is in line with the report that the performance of self-supervised pretrained speech models is robust to phonological complexity.

Furthermore, Figure 3 showed in detail that clicks are recognized relatively better and more stably than the other manners of articulation. Vowels exhibited relatively higher error rates in both languages because vowel quality distinctions often form more gradient acoustic continua, which may make them more difficult for ASR models to distinguish reliably.

Conclusion

The study concluded that these models consistently yielded lower phoneme error rates for click consonants than for nonclick phonemes. This finding suggests that the self-supervised training paradigm of these speech models exhibits strong adaptability to speech sounds unseen in the pretraining data.

The material was based on work supported in part by the US National Science Foundation under Grant Number BCS-2109709 and IIS-2137396 and by the JSPS KAKENHI Grant Number 22H04929, 22K18249, 23K25318, and 22K00536. We thank Florian Lionnet Alena Witzlack-Makarevich for the West!Xoon data. The first author of the paper, whose first language is not English, used generative AI tools to check grammar and improve the flow of the writing. Code autocompletion tools were also used for building the experimental code. All AI-generated content was carefully reviewed by the author and by native English speakers. The authors take full responsibility for the content of this paper. The material was based on work supported in part by the US National Science Foundation under Grant Number BCS-2109709 and IIS-2137396 and by the JSPS KAKENHI Grant Number 22H04929, 22K18249, 23K25318, and 22K00536. We thank Florian Lionnet Alena Witzlack-Makarevich for the West!Xoon data. We are also grateful to the reviewers and the metareviewer of Interspeech 2026 for providing constructive feedback. The material was based on work supported in part by the US National Science Foundation under Grant Number BCS-2109709 and IIS-2137396 and by the JSPS KAKENHI Grant Number 22H04929, 22K18249, 23K25318, and 22K00536. We thank Florian Lionnet Alena Witzlack-Makarevich for the West!Xoon data. The first author of the paper, whose first language is not English, used generative AI tools to check grammar and improve the flow of the writing. Code autocompletion tools were also used for building the experimental code. All AI-generated content was carefully reviewed by the author and by native English speakers. The authors take full responsibility for the content of this paper. The material was based on work supported in part by the US National Science Foundation under Grant Number BCS-2109709 and IIS-2137396 and by the JSPS KAKENHI Grant Number 22H04929, 22K18249, 23K25318, and 22K00536. We thank Florian Lionnet Alena Witzlack-Makarevich for the West!Xoon data. We are also grateful to the reviewers and the metareviewer of Interspeech 2026 for providing constructive feedback. The material was based on work supported in part by the US National Science Foundation under Grant Number BCS-2109709 and IIS-2137396 and by the JSPS KAKENHI Grant Number 22H04929, 22K18249, 23K25318, and 22K00536. We thank Florian Lionnet Alena Witzlack-Makarevich for the West!Xoon data. The first author of the paper, whose first language is not English, used generative AI tools to check grammar and improve the flow of the writing. Code autocompletion tools were also used for building the experimental code. All AI-generated content was carefully reviewed by the author and by native English speakers. The authors take full responsibility for the content of this paper.

Improvements for AI systems

Bold Header: Fine-tuning self-supervised models for rare phonemes

The improved system can accurately recognize click consonants by fine-tuning pretrained self-supervised speech models (Wav2Vec 2.0 and HuBERT) on data from click-rich Khoisan languages like Gui and West !Xoon, which consistently recognize clicks more accurately than non-click phonemes.

Bold Header: Robust generalization across typologically rare sounds

The system will demonstrate robust generalization to typologically rare sounds because the fine-tuned models are shown to overcome the limitations of pretraining data skew, suggesting that self-supervised pretraining supports generalization beyond dominant language distributions.

Bold Header: Improved performance on click consonant recognition

The ASR system will achieve significantly lower error rates for click consonants when compared to non-click phonemes, as evidenced by the Wilcoxon signed-rank test showing that click consonants are recognized with significantly lower error rates (W = 0, p = 0.016).