When Does Predictor-Based RL Align with Human Perception? A Study of Subjective Rewards in Codec-Based Speech Language Models

arXiv:2608.31035 · cs.CL, cs.SD, eess.AS · Submitted 2026-08-31 · Read on arXiv

cs.CL, cs.SD, eess.AS

Submitted: 2026-08-31

Updated: 2026-08-31

Code: https://github.com/sizigi/animeGRPO.1

Project page: https://sizigi.github.io

Terminology

• GRPO — 76× in this paper · explained in 6 episodes
• Core — 59× in this paper · explained in 2 episodes
• Calibration — 16× in this paper · explained in 7 episodes
• Actor — 4× in this paper · explained in 3 episodes
• Alignment — 4× in this paper · explained in 1 episode
• Fidelity — 4× in this paper · explained in 2 episodes
• Fine-Tuning — 4× in this paper · explained in 5 episodes
• Group Relative Policy Optimization — 3× in this paper
• Group Relative Policy Optimization (GRPO) — 3× in this paper · explained in 5 episodes
• Reinforcement Learning — 3× in this paper · explained in 4 episodes
• Robustness — 3× in this paper · explained in 5 episodes
• Transformer — 3× in this paper · explained in 2 episodes
• Accuracy — 1× in this paper · explained in 1 episode
• Consistency — 1× in this paper · explained in 3 episodes
• Decode — 1× in this paper · explained in 2 episodes
• Diffusion Model — 1× in this paper · explained in 8 episodes
• Direct Preference Optimization (DPO) — 1× in this paper · explained in 9 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• KL Regularization — 1× in this paper · explained in 2 episodes
• Provenance — 1× in this paper · explained in 4 episodes
• Proximal Policy Optimization (PPO) — 1× in this paper · explained in 5 episodes
• Regularization — 1× in this paper · explained in 3 episodes
• Reward Function — 1× in this paper · explained in 2 episodes
• Scaling Laws — 1× in this paper · explained in 4 episodes

Sources

• DLPO: Diffusion Model Loss-Guided Reinforcement Learning for Fine-Tuning Text-to-Speech Diffusion Models
• High Fidelity Neural Audio Compression
• CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens
• Differentiable Reward Optimization for LLM based TTS system
• Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation
• Who Finds This Voice Attractive? A Large-Scale Experiment Using In-the-Wild Data
• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
• Group Relative Policy Optimization for Text-to-Speech with Large Language Models
• Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis
• AnimeScore: A Preference-Based Dataset and Framework for Evaluating Anime-Like Speech Style
• MLS: A Large-Scale Multilingual Dataset for Speech Research
• Multi-Reward GRPO for Stable and Prosodic Single-Codebook TTS LLMs at Scale
• Fine-Tuning Language Models from Human Preferences
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering