Confucius4-TTS: Transcript-Free Cross-Lingual Zero-Shot TTS with a Learnable Speaker Encoder

Confucius4-TTS is a multilingual zero-shot text-to-speech (TTS) system that supports 14 languages and performs both intra-lingual and cross-lingual reference cloning without requiring transcripts of audio prompts. The system follows a two-stage architecture, consisting of text-to-semantic (T2S) and semantic-to-acoustic (S2A) modules. The LLM-based T2S module uses a learnable speaker encoder to extract timbre features from self-supervised speech representations, and the conditional flow-matching S2A module converts the predicted semantic tokens into mel-spectrograms. The same model also supports continuation cloning when a reference transcript is available. Confucius4-TTS is trained on large-scale multilingual speech data. It achieves high intelligibility and speaker similarity on public benchmarks. On the CV3-Eval cross-lingual benchmark, Confucius4-TTS obtains an average WER of 3.73% across six directions. On our internal cross-lingual set, it achieves the best average overall rank in human evaluation among recent open-source and commercial systems.

The model is trained across 14 languages: Chinese, English, Japanese, Korean, German, French, Spanish, Indonesian, Italian, Thai, Portuguese, Russian, Malay, and Vietnamese. The training corpus comprises approximately 500k hours of multilingual speech, including real and synthetic speech, curated with a data processing pipeline involving source separation, denoising, voice-activity-detection-based segmentation, filtering, language identification, multi-system ASR filtering, and speaker clustering.

The T2S model is a 24-layer decoder-only Transformer with a hidden size of 1280. It uses a pre-trained LLM tokenizer, with the embedding table projected to the T2S hidden dimension by a lightweight MLP. An ECAPA-TDNN speaker encoder produces a speaker embedding from features of a w2v-BERT 2.0 encoder. Semantic tokens are extracted with a pre-trained MaskGCT semantic tokenizer. The S2A model is a conditional flow-matching decoder with a Diffusion Transformer backbone, rendering 80-dimensional mel-spectrograms from semantic tokens, T2S hidden states, a global embedding from a CAM++ speaker verification model, and a reference mel-spectrogram prompt; a pre-trained BigVGAN vocoder reconstructs the waveform.

The T2S module predicts semantic tokens conditioned on the target text and the speaker embedding, which a learnable speaker encoder extracts from the reference audio. The S2A module predicts a mel-spectrogram from these predicted semantic tokens, the T2S hidden states, a global speaker embedding, and a prompt mel-spectrogram. The vocoder then converts the mel-spectrogram into a waveform.

The speaker encoder extracts timbre features from the reference audio, transforming a variable-length reference into a fixed-size conditional vector. It is trained jointly with the T2S module so that the resulting representation matches the requirements of the synthesis task. The speaker encoder operates on self-supervised speech representations rather than raw mel-spectrograms, capturing both content and speaker information, enabling extraction of the speaker embedding without the reference transcript.

The T2S model is trained with the autoregressive cross-entropy objective, jointly optimizing the speaker encoder, the text projection, the semantic-token embedding, and the Transformer backbone, while the pre-trained text embedding table remains frozen. The T2S hidden states at the semantic-token positions are retained and passed to S2A, reducing the information bottleneck introduced by discrete quantization.

The T2S model supports two inference modes: reference cloning and continuation cloning. In reference cloning, the T2S input contains only the speaker embedding and the target text, allowing greater flexibility in prosody and style. Continuation cloning additionally conditions on the reference transcript and the reference semantic tokens, yielding higher speaker similarity.

The S2A model is trained with conditional flow matching along an optimal transport path, with a DiT backbone estimating the vector field from the noisy input, the timestep, the frame-level condition, and the prompt mel-spectrogram. Classifier-free guidance is enabled by training the model on both conditional and unconditional situations, with a guidance strength of 0.7 at inference.

Experimental results show that Confucius4-TTS achieves top-tier cross-lingual intelligibility on CV3-Eval and remains among the leading systems on intra-lingual and multilingual zero-shot benchmarks. On CV3-Eval, Confucius4-TTS achieves the lowest error rate on four of the six directions and remains close to the best system on the other two. On X-Voice, Confucius4-TTS obtains the lowest CER on four directions and is within 0.3 absolute points of the best system on the remaining directions. On Seed-TTS-eval, reference cloning achieves 1.49 WER and 0.700 SIM on English, and 0.94 CER and 0.765 SIM on Chinese. On MiniMax-MLS-Test, reference cloning obtains the lowest WER on German and Thai and achieves speaker similarity above 72% across all evaluated languages.

In human evaluation on an internal cross-lingual test set covering zh→en, en→zh, zh→ko, and zh→ja, Confucius4-TTS ranks first or second in timbre similarity and naturalness on all four language pairs. It achieves the best overall rank on en→zh, zh→ko, and zh→ja; VoxCPM2 ranks first on zh→en. On pronunciation authenticity, Confucius4-TTS ranks first or second on all four language pairs, and first or second in emotional expressiveness on three of the four.

The main contributions are: Confucius4-TTS, a multilingual zero-shot TTS system that clones an unseen speaker in 14 languages without a reference transcript; a jointly trained speaker encoder that encodes reference audio into a speaker embedding for the T2S module; two inference recipes with the same model, enabling transcript-free reference cloning by default and continuation cloning when a reference transcript is available; and objective and subjective evaluations against open-source and closed-source systems, on which Confucius4-TTS performs competitively, with an open-source release of code, model checkpoints, and demos.

Improvements to AI systems:

1. Transcript-free speaker cloning in TTS: Implement a jointly trained learnable speaker encoder that extracts timbre features directly from self-supervised speech representations (e.g., w2v-BERT 2.0), eliminating the need for reference transcripts. This enables zero-shot voice cloning across 14 languages with high intelligibility (3.73% average WER on cross-lingual benchmarks).

2. Two-stage semantic-to-acoustic generation with reduced information bottleneck: Use an LLM-based text-to-semantic (T2S) module that passes hidden states (not just discrete semantic tokens) to a conditional flow-matching semantic-to-acoustic (S2A) module. This preserves prosodic and stylistic details, improving naturalness and speaker similarity in synthesized speech.

3. Dual-mode inference for flexible cloning: Support both reference cloning (transcript-free, more prosodic freedom) and continuation cloning (with reference transcript and semantic tokens, higher speaker similarity) using the same model. This allows users to trade off flexibility vs. fidelity based on available data.

4. Large-scale multilingual data curation pipeline: Adopt the described preprocessing pipeline—source separation, denoising, VAD-based segmentation, language identification, multi-system ASR filtering, and speaker clustering—to build a 500k-hour training corpus from real and synthetic speech. This improves robustness and cross-lingual generalization in any speech model.

5. Conditional flow matching with classifier-free guidance for acoustic modeling: Replace autoregressive vocoders with a Diffusion Transformer (DiT) backbone trained via optimal transport flow matching, using a guidance strength of 0.7. This yields faster, higher-quality mel-spectrogram generation with better speaker embedding conditioning (CAM++ global embedding + prompt mel).

6. Cross-lingual zero-shot adaptation without fine-tuning: Leverage the model’s ability to clone unseen speakers across language pairs (e.g., zh→en, en→zh, zh→ko, zh→ja) by conditioning on a single reference audio, achieving top ranks in human evaluation for timbre similarity, naturalness, pronunciation authenticity, and emotional expressiveness—without any per-language adaptation.

What the improved AI system can do:

• Clone an unseen speaker’s voice in 14 languages (Chinese, English, Japanese, Korean, German, French, Spanish, Indonesian, Italian, Thai, Portuguese, Russian, Malay, Vietnamese) using just a few seconds of untranscribed audio.

• Generate speech with near-human intelligibility (WER 0.70) in both intra-lingual and cross-lingual settings.

• Switch between two modes: (a) transcript-free cloning for flexible prosody/style, and (b) continuation cloning (when transcript is available) for maximum speaker fidelity.

• Produce natural, emotionally expressive, and authentic pronunciation across diverse language pairs, outperforming many open-source and commercial systems in human evaluations.

• Scale to large, noisy, multilingual datasets by using the robust data curation pipeline, enabling training on hundreds of thousands of hours without manual annotation.

• Generate mel-spectrograms efficiently via flow matching, enabling real-time or near-real-time synthesis with a vocoder, suitable for interactive applications.

Recent advances in zero-shot text-to-speech (TTS) have substantially improved speech quality and voice cloning fidelity. However, many zero-shot TTS systems still depend on audio prompt transcripts at inference time. This dependency limits cross-lingual voice cloning, since in-the-wild reference audio is often untranscribed. In this technical report, we present Confucius4-TTS, a multilingual zero-shot TTS system that supports 14 languages and performs both intra-lingual and cross-lingual reference cloning without requiring transcripts of audio prompts. Confucius4-TTS follows a two-stage architecture, consisting of text-to-semantic (T2S) and semantic-to-acoustic (S2A) modules. The LLM-based T2S module uses a learnable speaker encoder to extract timbre features from self-supervised speech representations, and the conditional flow-matching S2A module converts the predicted semantic tokens into mel-spectrograms. The same model also supports continuation cloning when a reference transcript is available. Confucius4-TTS is trained on large-scale multilingual speech data. It achieves high intelligibility and speaker similarity on public benchmarks. On the CV3-Eval cross-lingual benchmark, Confucius4-TTS obtains an average WER of 3.73% across six directions. On our internal cross-lingual set, it achieves the best average overall rank in human evaluation among recent open-source and commercial systems. We release code, model checkpoints, and demos at https://github.com/netease-youdao/Confucius4-TTS.