Zero-Shot Lombard Speech Synthesis with Controllable Style Embeddings

A controllable text-to-speech (TTS) system capable of synthesizing Lombard speech for any speaker without requiring explicit Lombard data during training is presented, which addresses limitations in existing methods that rely on specific training datasets. The core contribution lies in leveraging style embeddings learned from a large, prosodically diverse dataset and manipulating them via Principal Component Analysis (PCA) to generate speech at desired Lombard levels.

The gist: Our method enables zero-shot Lombard speech synthesis by learning a mapping between plain and Lombard speaker embeddings and interpolating between them during inference.

Model Architecture and Baseline

The proposed system is built upon the F5-TTS as the baseline, which is a flow-matching-based TTS model known for its high-quality outputs and built-in duration control. The architecture incorporates components inherited from F5-TTS (blue blocks) alongside new modules (green blocks). To inject speaker information without relying on in-context learning, an ECAPA-TDNN encoder extracts 1024D style features from reference mel-spectrograms. These features are then used via FiLM conditioning to scale and shift the block outputs. Furthermore, to prevent leakage of speaker characteristics, the masked mel-spectrogram inputs are augmented with formant shifts.

Style Embedding Analysis and PCA Manipulation

The research analyzed the learned style embedding space using PCA on both Lombard and articulation datasets to identify correlations with prosodic attributes. Loudness, associated with vocal effort, was analyzed using the AVID corpus, revealing that the first two principal components and measured speech pressure level (SPL), indicating that the style encoder inherently captures loudness variation. Clarity, associated with speaking rate and articulation, was analyzed using the ALBA dataset; PCA showed that the second principal component (PC2) strongly correlates with clarity. The control mechanism involves projecting style embeddings into this PCA space, shifting relevant components according to their variance and user-specified coefficients, and then applying an inverse PCA transformation to obtain manipulated embeddings.

Lombardness Control Mechanism

The system achieves fine-grained control over speech attributes through the manipulation of these style embeddings. The desired Lombard levels are simulated by setting specific parameters: Soft: loudness = -0.5, clarity = -0.5, speed = 1.0, Normal: loudness = 0.0, clarity = 0.0, speed = 1.0, Loud: loudness = 0.5, clarity = 0.5, speed = 0.9, and Very Loud: loudness = 1.0, clarity = 1.0, speed = 0.9. Additionally, the total duration of the output is adjusted by modifying the syllable-to-time ratio using F5-TTS's duration control to enhance perceived clarity during synthesis.

Evaluation and Performance Metrics

Experiments compared F5TTS-Style against F5TTS-Base on English and German prompts, measuring Word Error Rate (WER) for intelligibility, Speaker Similarity (SSIM) for speaker preservation, and UTMOS for perceived naturalness. In the cross-lingual setting with German prompts, F5TTS-Style substantially outperforms the baseline in both intelligibility (lower WER) and naturalness. Robustness to noise was assessed using relative WER (∆WER), which normalizes noisy WER by clean samples. The results demonstrated that control Lombardness enhances noise robustness, with ∆WER dropping sharply at higher Lombardness levels, confirming that intelligibility is preserved even under severe noise conditions (SNR = 1). Speaker identity preservation was confirmed by showing that absolute SSIM remains consistent across all Lombardness levels.

Conclusion and Contribution

The framework successfully enables zero-shot Lombard speech synthesis, generalization to unseen speakers, and direct control over prosody without additional Lombard data. The work introduces F5-TTS Style, a fine-tuned version of F5-TTS Base that supports cross-lingual speaker adaptation and explicit control of Lombard characteristics by leveraging large-scale prosodic style embeddings. This offers a robust solution for controllable Lombard TTS while maintaining speaker identity.

How it works

1. An ECAPA-TDNN encoder extracts 1024D style features from reference mel-spectrograms to generate the fixed-size style embedding used for conditioning.

2. The model incorporates these embeddings via FiLM conditioning in the later Diffusion Transformer blocks, where outputs are scaled and shifted by parameters derived from the style embeddings.

3. Lombardness is controlled by projecting style embeddings into a PCA space, shifting relevant components based on user-specified coefficients, and applying an inverse PCA transformation to obtain manipulated embeddings.

4. Duration control is applied at inference to adjust speech length based on the desired speaking rate, which is particularly important for simulating clear speech.

5. The system generates speech conditioned solely on the input text and the style embedding extracted from a reference audio sample, allowing for zero-shot synthesis without Lombard-specific training data.

Here are specific improvements for AI systems based on the proposed Lombard Speech Synthesis framework, along with what these improved systems can achieve:

) 1. Zero-Shot, Speaker-Agnostic Lombard Speech Synthesis:

The system can synthesize speech at any desired Lombard level (Soft, Normal, Loud, Very Loud) for a speaker whose voice is not present in the training data (zero-shot).

2. Controllable Prosodic Manipulation:

The system allows explicit modulation of two primary prosodic attributes—loudness and clarity—by shifting specific principal components (PC1 and PC2) of learned style embeddings via PCA manipulation.

3. Robustness to Acoustic Degradation (Noise Injection):

The improved system can generate speech that maintains high intelligibility even under varying levels of acoustic noise (SNR = 10, 5, 1), demonstrating superior performance compared to baseline models and ground-truth samples in terms of relative Word Error Rate (∆WER).

4. Speaker Identity Preservation Under Style Modulation:

The system ensures that manipulating Lombard attributes (loudness/clarity) does not degrade the perceived speaker identity, as evidenced by consistent and high Speaker Similarity Scores (SSIM) across all controlled Lombard levels.

5. Cross-Lingual Adaptability:

By leveraging style embeddings learned from a diverse dataset like Emilia, the system can generalize its control mechanism to generate Lombard speech for speakers in unseen languages (e.g., generating German Lombard speech based on an English speaker's style embedding).

6. Fine-Grained Duration Control:

The system incorporates duration control at inference, allowing users to adjust the speaking rate (syllable-to-time ratio) to further enhance perceived clarity, which is crucial for simulating clear speech under noise.

This improved AI system can be deployed in applications such as:

1. Hearing-Assistive Technologies: Providing real-time, high-quality Lombard speech synthesis for hearing aid users in noisy environments without needing personalized Lombard training data.

2. Robust Speech Recognition Systems (ASR): Enhancing the performance of Automatic Speech Recognition models by synthesizing Lombard versions of speech during inference to simulate noisy conditions, thereby reducing WER in real-world deployment.

3. Interactive Voice Assistants: Allowing AI assistants to dynamically adjust their output volume and articulation based on perceived environmental noise or the needs of a hearing-impaired interlocutor.

4. Speech Repair and Error Correction: Generating synthesized speech that compensates for distortions or artifacts in corrupted audio streams, making the resulting speech clearer for downstream processing.