DiffAnon: Diffusion-based Prosody Control for Voice Anonymization

arXiv:2604.26281 · eess.AS, cs.LG, cs.SD · Submitted 2026-04-29 · Read on arXiv

eess.AS, cs.LG, cs.SD

Submitted: 2026-04-29

Updated: 2026-08-28

Comments: Accepted to Interspeech 2026

License: http://creativecommons.org/licenses/by/4.0/

The gist: To preserve or not to preserve prosody is a central question in voice anonymization.

Terminology

Embeddings — 21× in this paper · explained in 2 episodes
Fidelity — 7× in this paper · explained in 2 episodes
Quantization — 4× in this paper · explained in 12 episodes
Generative AI — 2× in this paper · explained in 2 episodes
Generative Model — 2× in this paper · explained in 2 episodes
Iterative Refinement — 2× in this paper · explained in 3 episodes
Latent Representation — 2× in this paper · explained in 2 episodes
Verification — 2× in this paper · explained in 2 episodes
Accuracy — 1× in this paper · explained in 1 episode
Contrastive Learning — 1× in this paper · explained in 5 episodes
Diffusion Model — 1× in this paper · explained in 8 episodes
Diffusion Models — 1× in this paper · explained in 6 episodes
Diffusion Process — 1× in this paper · explained in 2 episodes
Distillation — 1× in this paper · explained in 2 episodes
Generative Models — 1× in this paper · explained in 2 episodes
Intelligence Advanced Research Projects Activity — 1× in this paper
Latent Diffusion Model — 1× in this paper · explained in 2 episodes
Latent Diffusion Models — 1× in this paper · explained in 2 episodes
Latent Representations — 1× in this paper · explained in 3 episodes
Office of the Director of National Intelligence — 1× in this paper
Pragmatics — 1× in this paper · explained in 2 episodes
Robustness — 1× in this paper · explained in 5 episodes
Self-Supervised Learning — 1× in this paper · explained in 8 episodes
Unified Framework — 1× in this paper · explained in 5 episodes

Abstract

To preserve or not to preserve prosody is a central question in voice anonymization. Prosody conveys meaning and affect, yet is tightly coupled with speaker identity. Existing methods either discard prosody for privacy or lack a principled mechanism to control the utility-privacy trade-off, operating at fixed design points. We propose DiffAnon, a diffusion-based anonymization method with classifier-free guidance (CFG) that provides explicit, continuous inference-time control over prosody preservation. DiffAnon refines acoustic detail over semantic embeddings of an RVQ codec, enabling smooth interpolation between anonymization strength and prosodic fidelity within a single model. To the best of our knowledge, it is the first voice anonymization framework to provide structured, interpolatable inference-time prosody control. Experiments demonstrate structured trade-off behavior, achieving strong utility while maintaining competitive privacy across controllable operating points.

DiffAnon: Diffusion-based Prosody Control for Voice Anonymization

Terminology

Abstract

Sources

Related papers