SITA: Learning Speaker-Invariant and Tone-Aware Speech Representations for Low-Resource Tonal Languages

arXiv:2601.09050 · cs.CL · Submitted 2026-01-14 · Read on arXiv

cs.CL

Submitted: 2026-01-14

Updated: 2026-09-15

Code: https://github.com/facebookresearch/fairseq

License: http://creativecommons.org/licenses/by/4.0/

The gist: Tonal low-resource languages are widely spoken but remain underserved by modern speech technologies.

Terminology

Abstract

Tonal low-resource languages are widely spoken but remain underserved by modern speech technologies. A central challenge is learning speech representations that are robust to nuisance variation, such as speaker gender, while preserving lexical tone, which carries word meaning. We propose SITA, a lightweight adaptation recipe for pretrained wav2vec-style self-supervised speech encoders. Rather than designing a new backbone or objective, SITA combines existing objectives in a staged optimization framework to reduce tone collapse while preserving ASR capability. Stage 1 improves speaker invariance without erasing tonal contrasts by combining a cross-gender contrastive loss with a tone-repulsive loss that separates same-word, different-tone realizations. Stage 2 restores recognition-oriented linguistic information through CTC fine-tuning and knowledge distillation on upper encoder layers. We evaluate SITA primarily on Hmong, a tonal language with limited digital resources and a small speaker pool. Against multilingual, speaker-adversarial, label-aware, and semi-supervised baselines, SITA achieves the best trade-off between cross-gender lexical retrieval and tone separation, while maintaining ASR accuracy close to an ASR-adapted XLS-R teacher. Results on Mandarin show consistent gains, suggesting that SITA is a general plug-in recipe for tonal speech representation learning.

Sources

Related papers