Your Voice Cloning System is Secretly a Voice Anonymizer

arXiv:2608.27360 · cs.CL · Submitted 2026-08-27 · Read on arXiv

cs.CL

Submitted: 2026-08-27

Updated: 2026-08-27

Code: https://github.com/rm00cr/coqui-tts

License: http://creativecommons.org/licenses/by/4.0/

The gist: Speaker anonymization suppresses speaker-identifying attributes from speech while preserving linguistic content and quality.

Terminology

Embeddings — 10× in this paper · explained in 2 episodes
Iterative Refinement — 8× in this paper · explained in 3 episodes
Cosine Similarity — 3× in this paper · explained in 3 episodes
Latent Space — 3× in this paper · explained in 7 episodes
Self-Supervised Learning — 3× in this paper · explained in 8 episodes
Transformer — 3× in this paper · explained in 2 episodes
Verification — 3× in this paper · explained in 2 episodes
Comparative Mean Opinion Score — 2× in this paper
Feature Extraction — 2× in this paper · explained in 2 episodes
IEEE Automatic Speech Recognition and Understanding Workshop — 2× in this paper
Robustness — 2× in this paper · explained in 5 episodes
Adaptation — 1× in this paper · explained in 1 episode
Benchmark — 1× in this paper · explained in 2 episodes
Core — 1× in this paper · explained in 2 episodes
Domain Adaptation — 1× in this paper · explained in 5 episodes
Fidelity — 1× in this paper · explained in 2 episodes
Generalization — 1× in this paper · explained in 13 episodes
Latent Representation — 1× in this paper · explained in 2 episodes
Latent Representations — 1× in this paper · explained in 3 episodes
Low-Resource Languages — 1× in this paper · explained in 4 episodes

Abstract

Speaker anonymization suppresses speaker-identifying attributes from speech while preserving linguistic content and quality. We propose repurposing XTTSv2, a multilingual voice cloning model trained on 27k hours of speech, for speaker anonymization without retraining. Our key insight is that XTTSv2's voice cloning capabilities preserve prosodic structure independently of speaker identity, enabling voice conversion by conditioning on a pseudo-speaker. We introduce an iterative refinement strategy that balances privacy and utility by maximizing a harmonic mean of speaker dissimilarity and intelligibility. Evaluated on seven European languages across CommonVoice and Multilingual LibriSpeech, our system achieves near-optimal privacy (EER about 0.49), competitive intelligibility, and substantially better speech quality than dedicated anonymization baselines, while requiring no language-specific training. We release the code here: https://github.com/rm00cr/coqui-tts.

Your Voice Cloning System is Secretly a Voice Anonymizer

Terminology

Abstract

Sources

Related papers