When KL Regularization Misfires in Group Policy Optimization
cs.LG, cs.CL
Submitted: 2026-10-08
Updated: 2026-10-08
Terminology
Sources
- Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimization
- Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model
- AGPO: Adaptive Group Policy Optimization with Dual Statistical Feedback
- QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuning
- GoLongRL: Capability-Oriented Long Context Reinforcement Learning with Multitask Alignment
- A Comedy of Estimators: On KL Regularization in RL Training of LLMs
- sGPO: Trading Inference FLOPs for Training Efficiency in RLVR
- On a few pitfalls in KL divergence gradient estimation for RL
- What is the Alignment Objective of GRPO?
- Anchored Policy Optimization: Mitigating Exploration Collapse Via Support-Constrained Rectification
- AGPO: Asymmetric Group Policy Optimization for Verifiable Reasoning and Search Ads Relevance at JD
- Group Sequence Policy Optimization
- Group Orthogonalized Policy Optimization:Group Policy Optimization as Orthogonal Projection in Hilbert Space
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks