TTS-Q: Perception-Guided Post-Training Quantization of Text-to-Speech Diffusion Transformers

summary

Video file (mp4)

The gist

The gist TTS-Q proposes a speaker-aware post-training quantization method that combines coverage-guided calibration sampling with speaker identity distillation to improve speech quality in zero-shot

In short

TTS-Q addresses how post-training quantization degrades speaker quality in large text-to-speech models by introducing a speaker-aware method. It combines calibration sampling guided by speaker embeddings with identity distillation using a full-precision teacher's speech. This approach recovers high speaker similarity and naturalness even when using low 4-bit quantization, proving it preserves voice timbre without sacrificing intelligibility.

Key concepts

Post-Training Quantization (PTQ)
This is a technique used to shrink large AI models by reducing the precision of their numerical values after they have been fully trained. While it saves memory and speeds up inference, standard PTQ often causes quality loss in speech models because it doesn't account for how different numbers affect human hearing.
Coverage-Guided Calibration Sampling
This is a method for selecting the best audio samples to calibrate the model during quantization. Instead of picking random samples, this technique uses speaker embeddings to ensure that the selected conditions cover a wide range of speaker identities, minimizing errors across all voices within a set budget.
Speaker Identity Distillation
This is a secondary calibration step designed specifically to preserve the unique voice characteristics after quantization. It forces the quantized model's output to match the style and timbre of high-quality, full-precision speech generated by a teacher model, ensuring speaker identity is maintained.
W4A4 PTQ Study
This refers to a specific experimental setup where the 1B and 3.5B diffusion transformer models are quantized using only 4 bits for both weights and activations. The study investigates how this extreme compression affects speech quality, showing that TTS-Q successfully mitigates the resulting loss of speaker timbre.

Terminology used across episodes

This episode discusses

The paper

TTS-Q: Perception-Guided Post-Training Quantization of Text-to-Speech Diffusion Transformers · Read on arXiv

Wyze Inc. · City College of New York, City University of New York · Institute of Computing Technology, Chinese Academy of Sciences · National Taiwan University · Tsinghua University

Zero-shot text-to-speech (TTS) can clone a voice from seconds of audio. Yet the quality comes from billion-parameter diffusion transformers whose size and multi-step sampling put them out of reach of the on-device, assistive, and real-time applications that need them most. Post-training quantization (PTQ) shrinks such models without retraining, but it has been built for language and vision, not speech. At 4 bits, a strong quantizer leaves intelligibility close to full precision but consistently erodes speaker timbre, so error rates stay low while the voice drifts from the target speaker. We ask why quantization hurts speech, where the damage lives, and whether it can be repaired without retraining. To find out, we run the first 4-bit weight-and-activation (W4A4) PTQ study of a TTS diffusion transformer at 1B and 3.5B. The timbre cost comes from both the weight and the activation quantization, but only the activation part has the timestep structure a precision schedule can exploit. We therefore introduce TTS-Q, which selects hard-to-quantize items from task-matched calibration candidates and applies a budget-balanced mixed-precision schedule that assigns higher precision to late-step feed-forward activations. TTS-Q recovers timbre best at both scales, at no cost to intelligibility. Separately, the deployed calibrated-W4A4 reference reduces peak memory on GPU, providing a practical reference for resource-constrained TTS deployment.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "TTS-Q: Perception-Guided Post-Training Quantization of Text-to-Speech Diffusion Transformers".

Jane: The gist TTS-Q proposes a speaker-aware post-training quantization method that combines coverage-guided calibration sampling with speaker identity distillation to improve speech quality in zero-shot text-to-speech diffusion transformers.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So, we're talking about this paper now called "TTS-Q: Perception-Guided Post-Training Quantization of Text-to-Speech Diffusion Transformers." It sounds a bit technical, but the core idea is that when you compress these big text to speech models—these diffusion transformers—you have to make sure you don't just lose the words, but also not lose who is speaking.

Jane: Exactly. The authors are Xiaoqin Feng, Yuqi Li, Weilun Feng, Yi-Cheng Lin, Kai Li, Xianglong Wang, Kuo Yang and Chuanguang Yang. What they're tackling here is the problem that post-training quantization has to solve: you need to keep both the clarity of the speech and the specific voice identity.

Lu: The paper sets up this challenge by asking two main questions for calibration: which reference voices should we use, and how do we actually preserve that voice identity when we quantize the model? It's about picking which voices matter most and then figuring out a way to lock in the speaker's characteristics during the compression process.

Meng: If I look at this from a deployment side, what I need to know is how much extra effort this adds during that post-training phase. Does it make things significantly slower than just doing standard quantization?

Lalam: The gist of what they propose is TTS-Q, which is a speaker-aware post-training quantization method that combines coverage-guided calibration sampling with speaker identity distillation >

The paper's summary: Tom: Okay, so the core mechanism involves two main parts. First, they use coverage-guided calibration sampling to choose the right conditions for reconstruction. They look at the speaker embedding space and select reference conditions that give good coverage under a fixed calibration budget F >

Jane: So, how do those conditions relate to quality? The paper says they measure distance in this speaker embedding space using something like farthest-point or average-distance cost under a fixed budget K >

Lu: They define the distance r of a sample S to the nearest selected condition as r(S) equals the minimum d of i in P, where d is max(zero one minus u i transpose u j) >

Meng: That sounds like they are trying to find voices that are well-represented in the embedding space before they start quantizing. What's the actual benefit there? Is it just better fidelity on average?

Lalam: They use two geometric objectives for cost minimization: Cmax(S), which emphasizes the least covered reference embedding, and Cavg(S), which gives more weight to densely populated regions >

The paper's improvements: Tom: Then they move to the second stage, speaker identity distillation. This is where they use the full-precision teacher's synthesized speech as an identity target for the student model. They minimize a cosine dissimilarity loss Lid(c; ϕ) = D E(Geϕ(c), sg

E(G(c)): ) >

Jane: That loss function is key because it’s not just comparing the final audio waveforms directly. It compares the speaker embeddings generated by both the teacher and student after they've gone through their respective generators >

Lu: This means they are trying to preserve voice identity without needing perfect sample-wise waveform agreement, which is a big step forward since we don't always have access to those perfect teacher states >

Meng: So, this is about making sure the student model learns the *style* of the voice from the teacher's full precision output rather than just matching every single tiny detail in every reconstruction unit >

Lalam: They combine these two into one final calibration objective: Lcal(c; ϕ) = Lrec(c; ϕ) + lambda Lid(c; ϕ), where lambda is a non-negative weight that controls how much they care about the identity term over the reconstruction loss >

Conclusion: Tom: So, to wrap up, TTS-Q uses coverage-guided calibration sampling to pick good reference conditions in speaker embedding space and then adds speaker identity distillation to supervise the quantization parameters using a weighted loss function >

Jane: The practical outcome they show is that this method consistently achieves higher speaker similarity SIM and predicted naturalness UTMOS than external baselines on W4A4 >

Lu: They found that the selection strategy matters, for instance, farthest-point selection performed better than average-distance selection on all reported metrics for LongCat-1B > <ref:2610.12474#pg1>

Meng: From a deployment standpoint, they also show some speed improvements. For LongCat models, it runs one point six six to two point three four times faster during warm inference when using INT8 execution on H100s >

Lalam: And finally, they noted that the choice of calibration objective changes depending on the model; farthest-point selection was chosen for LongCat-1B while average-distance selection was used for LongCat-3 point 5B and F5-TTS based on development set speech quality >

More episodes

← Home