Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL

arXiv:2608.11669v1 [cs.LG] 12 Aug 2026

---

Reinforcement learning against rubrics—lists of criteria graded by an LLM judge—has become a standard way to post-train language models on tasks with no deterministic answer. However, the paper identifies a fundamental weakness: "A rubric is a proxy for quality, not quality itself, and it is a fixed proxy. The same criteria are scored at every training step, many of them generic templates that repeat across prompts ('uses clear language', 'well-organized'). This immutability makes the proxy exploitable: A criterion rewarded identically at every step is a stable target. Once the policy finds a cheap, surface-level way to satisfy it, the shortcut is reinforced at every subsequent step."

The paper identifies three problems standing in the way of treating reward hacking seriously:

1. Measurement — detecting hacking requires a quality estimate independent of the training judge and training rubrics, namely OOD prompts and rubrics graded by a stronger cross-family judge.

2. Mitigation — the only rubric-specific approach known (POW3R reweighting) is untested for hacking and is found to hurt.

3. Compatibility with GRPO — any scheme perturbing the reward per step must respect group-relative RL; if rollouts of one prompt are graded on different criteria, advantages become incomparable and the gradient is corrupted.

4. An in-loop, two-judge protocol for measuring OOD reward hacking in rubric RL, and the first demonstration that the standard recipe reward-hacks out of distribution, on two independent benchmark pairs.

5. Rubric Dropout, a one-line, judge-cost-free regularizer against rubric reward hacking, with group-shared masking that makes it sound under GRPO.

6. Evidence that it works on both pairs, with higher gold at every matched checkpoint at 8B, higher window means at both sizes, and lower hacking on two independent measures, at no in-domain cost.

7. Ablations mapping the design space, showing a usable 30–50% range for the dropout fraction and evidence that criterion reweighting (POW3R-style) backfires.

For a query x and response y, a rubric is a set of K criteria indexed by k, each with weight w k. A single judge call grades all criteria at once, returning verdict s k(x,y) ∈ 0,1. The standard reward is:

R(x,y) = clip[0,1] (Σ k w k s k(x,y)) / (Σ k w k)

The protocol uses two judges and an OOD evaluation set. Every 20 training steps, the current policy is evaluated on the OOD evaluation set and graded twice—once with the training (proxy) judge and once with a stronger, cross-family (gold) judge. Four quantities are tracked:

• gold score: the gold judge's score on the OOD evaluation set

• proxy−gold gap: how much the proxy judge over-rates the policy

• overclaim fraction: the share of criteria the proxy marks satisfied but gold rejects

• in-domain full-rubric reward: what training itself is optimizing

The key insight: A judge with a fixed bias shifts a curve by a constant. What a fixed bias cannot do is make the gold curve fall while the proxy curve rises. Divergence between the two curves during training is the hacking signal.

The method has a single hyperparameter, the dropout fraction f ∈ [0,1). At each training step, a random f-fraction of the rubric's positive-weight criteria is dropped (always keeping at least three), and the reward is computed on the kept criteria only:

R̃(x,y;m) = (Σ k m k w k s k(x,y)) / (Σ k m k w k)

Dropout never touches a protected set reserved for safety-critical criteria, and evaluation always scores the full rubric. Since the judge grades all K criteria in one call anyway, the full-rubric reward stays available for logging at no extra cost.

One mask is drawn per rollout group, so all rollouts of a prompt at a given step are scored on the same sub-rubric. The mask RNG is seeded with SHA256(instance id, step), requiring no cross-worker communication and being reproducible.

The paper proves two key results:

• Proposition 1 (The normalizer cancels): Because the mask is shared by the whole group, any positive normalizer Z that depends only on the mask is the same constant for every response in the group, so it cancels in the standardized advantage.

• Observation 1 (Dropout is a variance regularizer): Over the i.i.d. mask distribution, E m[u i(m)] = (1−f)u i(1), and Var m[u i(m)] = f(1−f)Σ k w2 k δ2 k,i. The variance term is largest exactly when the advantage hinges on one high-weight criterion (one large w k δ k,i), and smallest when a response is broadly better than its group.

• Models: Qwen3-8B (primary) and Qwen3-4B (second scale), trained with GRPO (16 rollouts per prompt, learning rate 10−6)

• Train→eval pairs: RubricHub-Medical → HealthBench-Hard (1,000 prompts, physician-written rubrics) and RubricHub-Science → ResearchQA (368 validation prompts never occurring in training)

• Judges: proxy = gpt-4o-mini, gold = claude-sonnet-4-6

• Primary comparison: base (no dropout) vs. 30% vs. 50% dropout

• Horizon: common 600-step horizon, fixed comparison window (steps 400–600), matched-checkpoint win counts

On the Medical pair (Fig. 2): In the first phase, proxy and gold rise together... Then, around step 240, gold peaks at 31.2% and starts to slide while the proxy continues to 72%. The proxy−gold gap widens from 29% to as much as 44%. On the Science pair: gold falling 22 points from its peak within 600 steps. This matches the over-optimization signature established by Gao et al. for learned reward models.

• Medical (8B): Both dropout runs exceed base's gold score at all 11 matched checkpoints in the window, with window means of +1.0 points at f=30% and +2.0 points at f=50%. The gain also comes at no in-domain cost, since all three runs, dropout included, reach at least 97% in-domain full-rubric reward.

• Science (8B): "The base run's gold score falls from a peak of ∼67% to ∼46% by step 600, a 21.5-point decline, whereas the dropout runs give back 18.0 and 16.4 points of theirs... They exceed base at every matched checkpoint, with window means of +6.4 and +7.0 points at f=30% and f=50%."

• 4B results: Peaks stay near-tied, both dropout runs improve the window gold score (+0.7 to +5.3 points), and the in-domain full-rubric reward stays matched.

In every panel both dropout runs end the window below base on both measures (window means), at 8B by roughly 2–3 points on Medical and by nearly 8 points on Science. At 4B, base's window gap and overclaim near 47% on both pairs.

At matched proxy pass rates (within 1.3 points everywhere), both dropout runs have a higher gold pass rate and less overclaim, and both improve monotonically with the dropout fraction, up to +3.6 points of gold pass rate on Medical and +7.3 on Science at f=50%. The gains concentrate on expensive criteria: on Medical, the clinical axes (accuracy, completeness, context-awareness) rather than communication ones; on Science, analytical types (comparison, limitation, impact) gain two to three times as much as example and generic ones.

Sweeping f ∈ 20, 30, 40, 50, 60 %: "Best-checkpoint gold is essentially tied across all runs, from 30.6% to 31.5% against base's 31.2%... Everything from 20% to 50% is at or above base, with the best window mean at 50% (+2.0)... Only at 60% does the sign flip (−0.5), which is the expected failure mode. Drop too much and the surviving sub-rubrics stop covering what quality means."

"The natural alternative to dropping criteria, reweighting toward the informative ones, performs worse out of distribution than no intervention at all. POW3R attains the lowest OOD gold score of any run (27.0%), loses to base at all 11 matched checkpoints, and posts the highest overclaim fraction, 42.2%, above even base's 40.4%. The paper notes POW3R's best checkpoint (31.0%) matches base's (31.2%), so peak capability is intact. The deficit is in the decay that follows."

The paper acknowledges two possible mechanisms: "The motivating story is anti-co-adaptation. With the rubric resampled every step, no fixed criterion is reliably present to be gamed. A more boring story is implicit regularization. Dropout adds gradient noise, training moves more slowly along the same path, and the policy simply arrives at the hacking regime later. Both stories predict the same figures in this paper."

The decisive test—the gold-versus-overclaim frontier at two-plus epochs—is left to future work: Separation would establish the co-adaptation mechanism, and continued overlap would mean the gains reduce to implicit early stopping.

• Single seed: Every configuration is a single training run, because preemptible-only compute ruled out seed replication.

• Gold judge is not ground truth: A stronger judge is still a judge. Our claims rest on divergence and on run-to-run comparisons under identical judges, both of which survive a constant judge bias.

• In-domain cost measured on training set only: It does not rule out a small cost on unseen in-domain prompts, which we did not measure.

• Scope: One policy family at two sizes (Qwen3-8B and-4B), two domains, one RL algorithm (GRPO).

"Rubric-as-reward RL optimizes a fixed, imperfect proxy, and we showed that it does what fifty years of Goodhart warnings predict. Out of distribution, on two unrelated benchmark pairs, true quality peaks and then declines while the proxy score keeps rising. Rubric Dropout is the cheapest intervention we know of. It costs one line, one hyperparameter, and no extra judge calls. It raised the OOD gold score at every matched checkpoint in both domains at 8B, raised the window means at both model sizes, cut both of our hacking measures, and cost nothing on in-domain training prompts. Its hyperparameter has a wide safe range. The opposite design, criterion reweighting, made things worse."

Based on this paper, here are specific improvements to AI systems:

• Implementation: Modify the reward computation in GRPO/PPO to randomly drop 30–50% of rubric criteria per training step, with masks shared across all rollouts in a group (seeded by instance id + step).

• Benefit: Prevents the policy from overfitting to fixed rubric criteria, reducing OOD reward hacking by 2–8 points on gold judge scores while maintaining in-domain performance.

• System capability: The AI will maintain true quality on unseen, out-of-distribution tasks rather than gaming surface-level rubric features.

• Implementation: During training, every 20 steps evaluate the current policy on a held-out OOD set using both the training judge (proxy) and a stronger cross-family judge (gold). Track the proxy−gold gap and overclaim fraction as early-warning signals.

• Benefit: Provides real-time detection of reward hacking before it degrades final model quality. The system can halt or adjust training when the proxy and gold curves diverge.

• System capability: The AI can self-monitor for over-optimization and stop training at the true quality peak rather than the proxy peak.

• Implementation: If a system currently uses POW3R-style reweighting (emphasizing informative criteria), replace it with Rubric Dropout.

• Benefit: The paper shows reweighting backfires (lowest OOD gold score, highest overclaim fraction), while dropout consistently improves OOD robustness.

• System capability: The AI avoids the degradation pattern where peak capability is intact but quality decays rapidly after the optimum.

• Implementation: During evaluation, compute per-criterion gold pass rates and overclaim rates to identify which rubric axes are most gameable. Use this to dynamically adjust dropout fractions per criterion type (e.g., higher dropout on generic criteria like clear language, lower on clinical/analytical ones).

• Benefit: The paper shows gains concentrate on expensive, domain-specific criteria (accuracy, completeness, comparison, limitation) rather than generic ones. Targeted dropout can further improve this.

• System capability: The AI develops deeper, more substantive quality improvements rather than shallow surface-level compliance.

• Implementation: Use the paper's Observation 1 to compute the variance contribution of each criterion to the advantage estimate. When variance is dominated by a single high-weight criterion, increase the dropout fraction for that step to force broader improvement.

• Benefit: Reduces the risk of the policy exploiting one easy win criterion while ignoring others.

• System capability: The AI produces more balanced, multi-dimensional quality improvements rather than optimizing a single axis.

• Implementation: Reserve a protected set of criteria (e.g., safety, harmlessness, factual accuracy) that are never dropped, while applying dropout only to quality/communication criteria.

• Benefit: Ensures that dropout never compromises safety-critical requirements, while still preventing overfitting on non-critical rubrics.

• System capability: The AI maintains guaranteed safety standards while improving general quality robustness.

• Implementation: Use dropout runs' gold score trajectories to identify the true quality peak. Compare with base runs to determine if gains are due to anti-co-adaptation or implicit early stopping. If the latter, use dropout as a cheaper alternative to early stopping.

• Benefit: Reduces training compute by avoiding the need for extensive OOD evaluation to find the stopping point.

• System capability: The AI reaches optimal quality with fewer training steps and less computational overhead.