RLVR squared: Reinforcement Learning with Verifiable Rubric-based Ranking
cs.LG, cs.AI
Submitted: 2026-09-20
Updated: 2026-09-20
Comments: Preprint
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: Reinforcement Learning with Verifiable Rewards (RLVR) is expanding from tasks with well-defined correctness signals, such as mathematics and code, toward multifaceted quality requirements specified
Terminology
Abstract
Reinforcement Learning with Verifiable Rewards (RLVR) is expanding from tasks with well-defined correctness signals, such as mathematics and code, toward multifaceted quality requirements specified by multi-dimensional rubrics. Since policy optimization consumes one scalar per rollout, rubric-based pipelines must map multiple criterion scores into a scalar reward. This aggregation is often treated as score scaling, but it implicitly determines how quality dimensions trade off during training. The prevailing practice, normalizing each criterion and taking a linear combination, assumes that cardinal score differences are comparable across criteria and that gains on one criterion compensate for failures on another; both assumptions are unreliable when criteria are semantically heterogeneous. We propose Reinforcement Learning with Verifiable Rubric-based Ranking (RLVR squared), a verifiable ranking paradigm for rubric-based RLVR. For each criterion, RLVR squared converts rubric scores into criterion-specific within-group ordinal outcomes, recovers a latent utility from the resulting comparison matrix, and merges these utilities into one training signal. By retaining only within-group ordering and discarding raw score magnitudes, RLVR squared avoids calibrating heterogeneous rubric scales. It further supports objective-preserving attribute adjustment: auxiliary attributes that correlate with observed rankings but are not training objectives can enter the estimation without expanding the rubric or rewarding them directly. Across three model scales and 16 benchmarks, RLVR squared consistently outperforms representative rubric-based baselines, achieving the best overall performance on most benchmarks at every scale. Analysis shows it controls systematic effects tied to reasoning efficiency and response formatting while preserving the quality objective.
Sources
- HealthBench: Evaluating Large Language Models Towards Improved Human Health
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Reward Shaping to Mitigate Reward Hacking in RLHF
- Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
- SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge
- Focal Reward: Balanced Reinforcement Learning under Rubric-Based Rewards
- Reinforcement Learning with Rubric Anchors
- Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric
- Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation
- The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback
- From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
- GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization
- Mitigating False Credit Propagation: Probabilistic Graphical Reward Aggregation for Rubric-Based Reinforcement Learning
- From Absolute to Relative: Rethinking Reward Shaping in Group-Based Reinforcement Learning
- The Max-Min Formulation of Multi-Objective Reinforcement Learning: From Theory to a Model-Free Algorithm
- SysBench: Can Large Language Models Follow System Messages?
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark
- Proximal Policy Optimization Algorithms
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- The Trickle-down Impact of Reward (In-)consistency on RLHF
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks