Where the Verifier Fails: A Category-Level Audit of Reward Signals in RLVR

arXiv:2609.01354 · cs.CL, cs.LG · Submitted 2026-09-01 · Read on arXiv

cs.CL, cs.LG

Submitted: 2026-09-01

Updated: 2026-09-01

Code: https://github.com/ethxin0011/verifier-error-budget

Terminology

• Verifier — 63× in this paper · explained in 2 episodes
• Accuracy — 10× in this paper · explained in 1 episode
• Benchmark — 8× in this paper · explained in 2 episodes
• Reinforcement Learning — 5× in this paper · explained in 4 episodes
• Verification — 4× in this paper · explained in 2 episodes
• Robustness — 3× in this paper · explained in 5 episodes
• Verifiable Rewards — 2× in this paper · explained in 2 episodes
• Benchmarking — 1× in this paper · explained in 4 episodes
• Failure Modes — 1× in this paper · explained in 3 episodes
• Reward Hacking — 1× in this paper · explained in 8 episodes

Sources

• Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Models
• Faults in Our Formal Benchmarking: Dataset Defects and Evaluation Failures in Lean Theorem Proving
• Metamorphic Testing of Deep Code Models: A Systematic Literature Review
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers
• Training Verifiers to Solve Math Word Problems
• Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• A Single Character can Make or Break Your LLM Evals
• When the Reward Suite Is Leaky: A Preregistered Causal Contrast of Natural Verifier False Positives in RLVR
• LGMT: Logic-Grounded Metamorphic Testing for Evaluating the Reasoning Reliability of LLMs
• Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs
• From Accuracy to Robustness: A Study of Rule- and Model-based Verifiers in Mathematical Reasoning
• METAL: Metamorphic Testing Framework for Analyzing Large-Language Model Qualities
• Beyond F1: Evaluating Coverage and Failure Recovery in AI Model Security Scanners
• Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations
• Auditing Reward Hackability in Code RL Training Environments
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering