From Script to Drama: An Agentic Framework for Controllable Multi-Speaker Dialogue TTS

arXiv:2609.33362 · eess.AS, cs.CL, cs.SD · Submitted 2026-09-27 · Read on arXiv

eess.AS, cs.CL, cs.SD

Submitted: 2026-09-27

Updated: 2026-09-27

Project page: https://xkx-hub.github.io/FStD-demo/Abstract

Terminology

• Benchmark — 22× in this paper · explained in 2 episodes
• Consistency — 12× in this paper · explained in 3 episodes
• Ablation — 6× in this paper · explained in 1 episode
• Iterative Refinement — 6× in this paper · explained in 3 episodes
• Agentic Framework — 4× in this paper · explained in 6 episodes
• Verification — 4× in this paper · explained in 2 episodes
• Agentic System — 2× in this paper · explained in 2 episodes
• Alignment — 2× in this paper · explained in 1 episode
• Core — 2× in this paper · explained in 2 episodes
• Flow Matching — 2× in this paper · explained in 8 episodes
• Accuracy — 1× in this paper · explained in 1 episode
• Alternation — 1× in this paper · explained in 1 episode
• Benchmarking — 1× in this paper · explained in 4 episodes
• Cosine Similarity — 1× in this paper · explained in 3 episodes
• Diffusion Model — 1× in this paper · explained in 8 episodes
• Diffusion Models — 1× in this paper · explained in 6 episodes
• Embeddings — 1× in this paper · explained in 2 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Foundation Models — 1× in this paper · explained in 7 episodes
• Latent Diffusion Model — 1× in this paper · explained in 2 episodes
• Latent Diffusion Models — 1× in this paper · explained in 2 episodes
• Reinforcement Learning — 1× in this paper · explained in 4 episodes
• Robustness — 1× in this paper · explained in 5 episodes
• Transformer — 1× in this paper · explained in 2 episodes
• Two-stage training — 1× in this paper · explained in 2 episodes

Sources

• MINT-Bench: A Comprehensive Multilingual Benchmark for Instruction-Following Text-to-Speech
• An Enhanced Res2Net with Local and Global Feature Fusion for Speaker Verification
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training
• CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models
• Audio-Oscar: A Multi-Agent System for Complex Audio Scene Generation, Orchestration, and Refinement
• Qwen3-TTS Technical Report
• VoiceSculptor: Your Voice, Designed By You
• InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems
• Fish Audio S2 Technical Report
• VibeVoice Technical Report
• AuDirector: A Self-Reflective Closed-Loop Framework for Immersive Audio Storytelling
• OV-InstructTTS: Towards Open-Vocabulary Instruct Text-to-Speech
• Qwen3-ASR Technical Report
• Borderless Long Speech Synthesis
• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
• SoulX-Podcast: Towards Realistic Long-form Podcasts with Dialectal and Paralinguistic Diversity
• JoyTTS: LLM-based Spoken Chatbot With Voice Cloning
• VoxCPM2 Technical Report

Related papers

• X-VC: Zero-shot Streaming Voice Conversion in Codec Space
• Autoregressive Guidance of Deep Spatially Selective Filters using Bayesian Tracking for Efficient Extraction of Moving Speakers
• Anonymization, Not Elimination: Utility-Preserved Speech Anonymization
• Towards Audio Token Compression in Large Audio Language Models
• WaveScat: Wavelet Scattering Front-Ends with Self-Supervised Features for Speech Deepfake Detection
• ProPS: Prompted Profile Synthesis for Natural Language-Conditioned Speaker Embedding Distributions