HPRO: Hierarchical Progressive Reward Optimization via Preference Extraction for Emotional Text-to-Speech

Hierarchical Progressive Reward Optimization (HPRO) is a novel framework designed to enhance emotional expressiveness in Large Language Model-based Text-to-Speech (TTS) systems by overcoming structural mismatches between content and emotion optimization, and between sparse rewards and dense generation. The core finding is that HPRO significantly improves fine-grained emotional expressiveness while effectively preserving linguistic intelligibility through a hierarchical progressive optimization strategy guided by a novel differentiable reward model.

The Gist

HPRO proposes a hierarchical progressive reward optimization framework that introduces the HD-Emo codec as a differentiable reward model to resolve information conflict, and bridges the scale gap by progressively aligning frame-, word-, and sentence-level objectives, leading to enhanced emotional expressiveness while effectively preserving linguistic intelligibility.

How it works: Resolving Information Conflict via HD-Emo Codec

The framework first addresses the Information Conflict, where semantic content and emotion in a shared latent space produce conflicting gradients. To resolve this, HPRO introduces the HD-Emo codec as a novel differentiable reward model that projects speech tokens into distinct content and style preference subspaces. This mechanism structurally isolates stylistic optimization from semantic content. Specifically:

1. The codec processes discrete speech tokens through dual preference extractors with FSQ bottlenecks to obtain content and style preference tokens Tc and Ts.

2. To ensure strict semantic alignment, the latent representations prior to quantization are fed into a content adapter supervised by an ASR objective, formulated as the auto-regressive negative log-likelihood of target text tokens: "LASR = −∑j=1 log P(yj y<j, Tc) (1). Crucially, a stop-gradient mechanism" is applied to prevent acoustic leakage from influencing content extraction.

3. The style tokens are optimized via hierarchical supervision: at the sentence level, a pre-trained emotion2vec model supervises the predicted distribution pˆ from the emotion decoder through a CE loss (Equation 2). For fine-grained control, word-level constraints use metrics like the Concordance Correlation Coefficient (CCC) to enforce consistency across Valence–Arousal–Dominance dimensions.

How it works: Bridging the Scale Gap via Progressive Optimization

To bridge the Scale Gap between sparse sentence-level rewards and dense frame-level generation, HPRO employs a progressive optimization strategy that constructs a continuous gradient bridge. This is achieved by organizing supervision at three distinct levels:

1. Frame-level reward: This involves aligning generated pre-quantization representations (Zˆc, Zˆs) directly with ground-truth discrete preference tokens (Tc, Ts) using L1 regression losses: Lcp = ∥Zˆc − Tc∥1, Lsp = ∥Zˆs − Ts∥1 (6).

2. Word-level reward: This level incorporates boundary-aware constraints to refine local emotional trajectories. It utilizes a wVAD CCC loss LwV AD to match predicted wVAD trajectories with target signals, alongside the ASR loss (LASR) for semantic consistency.

3. Sentence-level reward: This ensures global affective alignment by aligning the predicted emotion distribution with target soft labels using a CE loss (LSER).

How it works: Progressive Optimization Strategy

The framework employs a three-stage progressive optimization strategy to ensure stable convergence and prevent reward hacking. The optimization is structured as follows:

1. Stage I (Frame-level warm-up): Initially, only frame-level preference alignment losses (Lcp and Lsp) and the KL regularization term are optimized, with initial weights set to λKL = 0.05, λsp = 2, and λcp = 1. The Gumbel temperature is initialized to τ = 2.

2. Stage II (Word-level refinement): Word-level supervision through LwV AD and LASR is introduced to refine local trajectories while preserving semantics, with weights adjusted to λKL = 0.02, λsp = 2, λcp = 1, λASR = 5, and λwVAD = 1. The Gumbel temperature is annealed to τ = 1.

3. Stage III (Sentence-level alignment): Finally, the sentence-level emotion classification loss (LSER) is incorporated with λSER = 0.5 and the temperature annealed to τ = 0.8, unifying the global affective style under comprehensive hierarchical supervision.

How it works: Experimental Validation

Experiments utilize LibriSpeech for ASR supervision and LSSED/EmoVoice-DB for emotional modeling, comparing HPRO against baselines like CosyVoice2 and HD-PPT. The objective metrics evaluated include MOSN (naturalness), MOS-E (consistency), WER, wVAD-CCC, EMO-SIM, and DNSMOS. The results demonstrate that HPRO achieves the highest MOSN and the second-highest MOS-E in subjective evaluation.

As a diligent AI researcher, I have analyzed the HPRO framework presented in this paper. The core innovation lies in its hierarchical, progressive reward optimization designed specifically to solve the information conflict and scale gap challenges inherent in preference-driven emotional Text-to-Speech (TTS).

Here are the specific improvements you can implement by adopting or extending the HPRO methodology, and what these improved AI systems can achieve:

) Hierarchical Content/Style Decoupling via HD-Emo Codec

The system should be upgraded from monolithic latent spaces to a structured preference space using the HD-Emo codec.

• A transformer backbone should be augmented with a dual preference extractor (Content Token Extractor and Style Token Extractor).

• Implement Finite Scalar Quantization (FSQ) to map continuous speech representations into discrete, separable tokens: Content Tokens (for semantic accuracy) and Style Tokens (for prosodic/emotional features).

• The system can achieve high fidelity in emotional synthesis while strictly guaranteeing linguistic intelligibility.

) Multi-Scale Gradient Bridging via Progressive Optimization

Instead of relying on single-scale rewards, the training objective must be structured across three distinct levels: Frame, Word, and Sentence.

• Implement a progressive optimization schedule: Start by optimizing frame-level alignment (dense acoustic structure), then refine with word-level constraints (temporal alignment + semantic integrity), and finally unify with sentence-level affective consistency.

• This allows the model to build a robust acoustic foundation before imposing global emotional constraints, effectively preventing early reward hacking.

• The system can generate speech where emotional nuances are captured at both the micro (word boundary) and macro (sentence context) levels, leading to highly coherent and expressive output.

) Fine-Grained Emotional Control via Hierarchical Loss Functions

The loss function must be explicitly multi-faceted, ensuring that different aspects of emotion are optimized independently yet coherently.

• Incorporate Content Loss: Use an ASR objective (LASR with stop-gradient mechanism) to strictly anchor the content tokens to the ground truth text, preventing semantic degradation.

• Incorporate Style Loss: Use hierarchical objectives—a global emotion distribution loss (LSER) for sentence context, and a fine-grained Valence–Arousal–Dominance (wVAD) constraint loss for word-level prosody alignment via CCC.

• The system can produce speech that is not only emotionally rich but also precisely controlled in terms of specific acoustic parameters like intensity, energy, and pacing.

) Robustness Against Reward Hacking via Structural Isolation

The HD-Emo codec ensures that the optimization targets are structurally isolated, mitigating the risk of optimizing for superficial emotional markers at the expense of linguistic quality.

• By decoupling content tokens from style tokens in the latent space, any gradient pushing for higher emotion is constrained to only affect style features, leaving content features largely protected by ASR supervision.

• The system can maintain high semantic accuracy (low WER) even when maximizing complex emotional scores, a critical capability that monolithic reward systems fail to provide.

This improved AI system can perform:

1. Generate highly expressive Text-to-Speech (TTS) output that captures complex human emotions with high perceptual quality (MOSN and MOS-E maximization).

2. Produce speech with superior linguistic intelligibility (low WER), ensuring that emotional modulation does not corrupt the underlying text meaning.

3. Achieve fine-grained prosodic control, allowing for precise manipulation of acoustic features like word boundary emphasis and sentence-level affective tone, surpassing the capabilities of stereotyped emotion models (like IndexTTS2).

4. Operate reliably in complex optimization landscapes, avoiding reward hacking by using a structured, progressive path that balances dense local acoustic alignment with global affective consistency.