Rethinking Speaker Embeddings for Speech Generation: Sub-Center Modeling for Capturing Intra-Speaker Diversity

arXiv:2407.04291 · eess.AS, cs.LG · Submitted 2024-07-05 · Read on arXiv

eess.AS, cs.LG

Submitted: 2024-07-05

Updated: 2026-08-28

Code: https://github.com/resemble-ai/Resemblyzer

Project page: https://lec-synt.github.io/sub-center-demo

Terminology

• Embeddings — 48× in this paper · explained in 2 episodes
• Verification — 12× in this paper · explained in 2 episodes
• Aggregation — 3× in this paper · explained in 1 episode
• Adam — 2× in this paper · explained in 2 episodes
• Clustering — 2× in this paper · explained in 2 episodes
• Cosine Similarity — 2× in this paper · explained in 3 episodes
• Distillation — 1× in this paper · explained in 2 episodes
• Fidelity — 1× in this paper · explained in 2 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Generative Model — 1× in this paper · explained in 2 episodes
• Personalization — 1× in this paper · explained in 2 episodes
• Regularization — 1× in this paper · explained in 3 episodes
• Representation Learning — 1× in this paper · explained in 2 episodes

Sources

• SpeechBrain: A General-Purpose Speech Toolkit

Related papers

• X-VC: Zero-shot Streaming Voice Conversion in Codec Space
• Autoregressive Guidance of Deep Spatially Selective Filters using Bayesian Tracking for Efficient Extraction of Moving Speakers
• Anonymization, Not Elimination: Utility-Preserved Speech Anonymization
• Towards Audio Token Compression in Large Audio Language Models
• WaveScat: Wavelet Scattering Front-Ends with Self-Supervised Features for Speech Deepfake Detection
• ProPS: Prompted Profile Synthesis for Natural Language-Conditioned Speaker Embedding Distributions