Adaptive Duration Model for Text Speech Alignment
summary
The gist
The gist In this paper, we propose an adaptive duration modeling framework called DurFormer to predict phoneme-level duration distribution with given text, which has shown more precise prediction and
In short
The DurFormer framework is a novel duration prediction model designed for speech-to-text alignment in neural TTS. It predicts phoneme-level durations using text input, speech speed, and scene conditions. This approach offers more precise alignment and better adaptation to different conditions than previous methods, improving zero-shot TTS robustness.
Key concepts
- Duration Prediction Framework
- This is the core idea of the paper: a system that predicts how long each phoneme in a text should last when spoken. It tries to find an optimal sequence of durations that best matches the text and speech features, serving as the foundation for accurate alignment.
- Attribute Encoder
- This component extracts and combines different types of information influencing speech duration. It separates factors into 'subjective' (like speaker characteristics) and 'objective' (like external speaking conditions) to create a richer understanding of how speech timing is affected.
- Alignment Objective
- The mathematical goal is to find the best mapping between phonemes and the resulting audio spectrogram. This objective is framed as a sequence prediction problem, where the model learns to predict the target duration sequence based on text and condition inputs.
Terminology used across episodes
This episode discusses
- Adaptive Duration Model for Text Speech Alignment · Paper Radio
- Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens
- Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis
- FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
- Seed-TTS: A Family of High-Quality Versatile Speech Generation Models
- NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models
- Prosody Transfer in Neural Text to Speech Using Global Pitch and Loudness Features
- Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
- MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer
- SimpleSpeech: Towards Simple and Efficient Text-to-Speech with Scalar Latent Transformer Diffusion Models
- F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching
- WenetSpeech4TTS: A 12,800-hour Mandarin TTS Corpus for Large Speech Generation Model Benchmark
The paper
Adaptive Duration Model for Text Speech Alignment · Read on arXiv
Junjie Cao
Tsinghua University
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: Today's paper: "Adaptive Duration Model for Text Speech Alignment".
Jane: The gist In this paper, we propose an adaptive duration modeling framework called DurFormer to predict phoneme-level duration distribution with given text,
Tom: First, who's behind it and why it matters.
Paper summary: Tom: So, to recap where we are, this paper proposes the Adaptive Duration Model for Text Speech Alignment. The thesis is that instead of just relying on fixed external sources for durations in non-autoregressive TTS, you can predict phoneme-level duration distribution directly from the text input itself >
Jane: Essentially, they claim their new framework offers a more precise way to align phonemes and mel spectrograms harmoniously by predicting those durations based on the text and context conditions >
Lu: The architecture is called DurFormer. It takes the text prompt, speed level, and speech scene as input. It uses an attribute encoder to fuse these features into enhanced tokens before feeding them into a prediction process >
Meng: I see they are using a semantic extractor block from Roberta for the text semantics, which means they’re trying to capture the meaning of the text deeply before predicting durations >
Tom: And then they incorporate that semantic vector through a cross-attention module and an MLP layer for domain adaptation, which feeds into their duration prediction mechanism >
Jane: The key claim is that this combination of linguistic features and contextual information allows the model to perform better than previous methods, specifically showing improvements on alignment accuracy and better robustness when dealing with audio mismatches >
Lalam: The objective is to find an optimal monotonic path s star of phi, which means searching through all possible alignment paths S(Phi) for the best fit against the target duration sequence >
Tom: So, if I'm driving and my voice sounds different than the training data, this model is supposed to be able to adapt its duration predictions better because it considers those conditions >
Lu: That’s right. And they quantify that improvement numerically, stating roughly an eleven point three percent improvement on alignment accuracy >
Meng: It's good that they gave a number like eleven point three percent, but I wonder if that number was tested on real-world traffic or just clean datasets > <ref:2507.22612#pg1>
Jane: That’s a valid concern, Meng. But the paper does claim it makes zero-shot TTS models more robust to the mismatch between prompt audio and input audio >
Tom: So they are showing that this adaptation capability isn't just theoretical; it translates into better performance when you try to use a model with minimal or no specific audio prompts >
Lalam: It’s all about learning how the linguistic characteristic of the text scripts affects the rhythm of the synthesized speech, which in turn dictates the duration of each phoneme >
Jane: So, if we're trying to understand this paper for someone who doesn't read it, it boils down to this: they built a system that uses text meaning and current speaking conditions to guess how long each part of a word should take when making speech from text >
Tom: It’s about moving beyond just using external sources for timing and embedding the actual text structure into the duration prediction itself. That's what they achieved in this paper.
Conclusion: Tom: So, looking at "Adaptive Duration Model for Text Speech Alignment" by Cao et al., the main point is that they’ve developed a novel duration prediction framework called DurFormer >
Jane: It moves beyond just using external sources for durations and instead learns to predict them directly from the text while incorporating contextual information like speaking speed level and scene >
Lu: The authors are trying to solve the alignment problem by searching for an optimal monotonic path s star of phi that aligns the phoneme and mel spectrogram precisely >
Meng: In simple terms, this means if you give it text and some conditions, it can give you a much better idea of how long each phoneme should be than before >
Tom: They show this translates to roughly an eleven point three percent improvement on alignment accuracy and better performance for zero-shot TTS models when there’s a mismatch in audio prompts >
Jane: So the implication is that for anyone using neural text to speech, especially those deploying systems where you can't rely on fixed timing data, having a model that adapts its duration based on context makes the system much more reliable.
Lu: They’re doing this by deeply fusing linguistic meaning into the prediction process through semantic fusion modules >
Lalam: It’s about learning how the text structure influences the rhythm of speech and adjusting those durations dynamically >
More episodes
- 2409.12500-LLMR: Knowledge Distillation with a Large Language Model-Induced Reward
- 2504.05607-Towards Reliable Long-Context Reasoning: Detecting Unanswerable Questions via FactGuard
- 2402.05804-InkSight: Offline-to-Online Handwriting Conversion by Teaching Vision-Language Models to Read and Write
- 2508.08100-Grid2Guide: A* Enabled Small Language Model for Indoor Navigation
- 2408.04140-UNLEARN Efficient Removal of Knowledge in Large Language Models
- 2504.03669-A self-learning framework for adaptive free-form pipe routing in aeroengines
- 2410.05203-Beyond FVD: Enhanced Evaluation Metrics for Video Generation Quality
- 2507.09227-PanoDiff-SR: Synthesizing Dental Panoramic Radiographs using Diffusion and Super-resolution
- 2510.03419-Multi-task Neural Diffusion Processes
- 2509.25146-Fast Feature Field (F cubed): A Predictive Representation of Events