2608.07341-Zero Gap Is Not Restoration: Stratified Per-Question Probability Evaluation and Step-wise Mitigation of Benchmark Contamination

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Zero Gap Is Not Restoration: Stratified Per-Question Probability Evaluation and Step-wise Mitigation of Benchmark Contamination".

Jane: The paper was written by Ruijie Hou, Yueyang Jiao, Zhao Wang and Yingming Li from Zhejiang University.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary: Tom: We've got a paper today that attacks something really insidious — benchmark contamination, where test questions leak into training data and quietly inflate a model's scores. The authors' headline claim is that the standard way of measuring whether a fix works is itself broken, so the field has been celebrating results that don't actually hold.

Jane: Yeah, and the title says it all — zero gap is not restoration. The old metric, G-AP, compares the average score of a contaminated model after mitigation against a clean model, and if the averages match, you declare victory. But that can happen while individual questions are still badly wrong — over-suppression on one cancels under-suppression on another.

Lu: So the paper builds a new metric called SA-PPG, which looks at each question's solve probability — the chance that a sample from the model gets that question right. It measures the gap per question, then groups questions by how likely the clean model is to solve them, and averages within groups.

Meng: And the grouping closes a real loophole. If you just average per-question gaps equally, a trivial strategy that makes the model fail everything scores well on the many questions the clean model can't solve anyway. Stratification is what stops that shortcut.

Jane: Then on the strategy side, there's RailCap. Instead of estimating which questions are contaminated in advance, it watches decoding in real time — whenever a sample falls back onto the greedy trajectory, it caps that trajectory token's probability to the runner-up, nudging generation away from memorized paths.

Tom: And the results are stark. Under the old metric, one existing method looks near-perfect — a gap of about 0 point 024. Under the new metric it falls to 0 point 293, barely better than doing nothing. RailCap, which wasn't even the best under the old metric, becomes the best under SA-PPG.

Lalam: That's the bigger point, I think. If your ruler is wrong, you're not just misreading results — you're designing toward the wrong target. The paper shows the whole contamination-mitigation line of work may have been rewarding strategies that don't actually restore capability, and the authors back that claim with numbers at every step.

Jane: Exactly, and the details really matter here — how the metric fails, how they fix it, and how RailCap works. The opening page gets straight to the first failure.

Page 1 of the paper: Tom: So the thesis is on the table — the ruler is broken and the fix has to come on two fronts. The first page digs into exactly why the old ruler misleads, and it comes down to two measurement problems.

Jane: The first problem is the readout. A single correct-or-incorrect mark tells you almost nothing about a single question, because sample the same question twice and you can get different answers. What stabilizes as you draw more samples is the solve probability — the chance a sampled response is right — and that's what should represent per-question performance, not a coin flip.

Lu: Then there's the order of aggregation. You average all the per-question readouts first, then take a difference between the mitigated and clean model. That order lets a question that's been over-suppressed — driven too far down — cancel out a question that's still inflated from memorization. The paper is very explicit that a zero gap can arise from pure cancellation.

Meng: So the title is literally the argument. Zero does not mean restored, because every question could be wrong in opposite directions. And the authors frame this as a measurement problem that shapes the whole field — how you define the metric determines how strategies get designed in the first place.

Tom: There's also the practical framing around why mitigation evaluation matters at all. Dataset-side fixes, like rebuilding or rewriting benchmarks, are costly, and the new data leaks again once released. Mitigation instead intervenes during decoding on datasets already at risk — no new data needed.

Jane: But you can't trust a mitigation strategy until you can measure it, and that's why the metric comes first in the paper. The authors are saying: fix the ruler before you celebrate the marks.

Lalam: And that's the part that should worry people in the broader community. Evaluation bugs in other fields have quietly redirected years of research. Here the authors are claiming that exact pattern is happening in contamination mitigation — methods tuned to produce cancellation because the metric rewards it. If they're right, a lot of prior conclusions need revisiting.

Tom: They set the whole trajectory for the paper from here — fix the readout and the aggregation, then build a strategy that survives the new ruler. The next page shows what happens when you try to fix those two flaws naively.

Page 2 of the paper: Jane: We left off with the diagnosis — discrete readouts and cancellation. Page two attempts the fix, and walking that path the authors stumble onto a brand new problem that nobody had seen coming.

Tom: The first corrected metric is A-PPG. You estimate each question's solve probability by sampling, take the absolute difference against the clean model per question, then average. The zero condition is strict — it reads zero only when every single question's solve probability matches the clean model. That kills the cancellation problem outright.

Lu: But then the equal-weighting trap appears. When the clean model is itself weak, a large share of questions have solve probability zero — on GSM8K with Llama-2, the clean model never solves nearly a quarter of the questions. So a trivial strategy that simply drives the contaminated model to fail on everything gets a perfect zero gap on that majority, and the harder questions get diluted away.

Meng: That's the loophole. The fix is stratification — group questions by the clean model's solve probability, average the gaps within each group, then average across groups. The zero-probability majority becomes one group among many rather than the whole story, and pushing everything to zero doesn't help you score well.

Lalam: And this is where the paper does something methodologically important — it doesn't just patch the loophole, it redesigns the aggregation so the loophole doesn't exist. That kind of fix holds up under scrutiny, and it changes which strategies look good. That's what makes the metric construction here a contribution on its own.

Jane: The page also has that really striking empirical picture — Figure 1. When a question is leaked, the contaminated model's sampled responses collapse onto its own greedy trajectory. On unleaked questions, the responses scatter across many different paths. So the sampling behavior itself is an online signal of memorization.

Tom: And the second observation matters just as much. At the decoding steps where the clean and contaminated models diverge, the token the clean model selects is, in about half the cases, the contaminated model's runner-up. That's a concrete hook for a mitigation strategy — if the clean model's token usually sits just below the top of the contaminated model's distribution, you can flatten that distribution and let it through.

Lu: So by the end of page two, the metric is complete and the behavioral clues are in place. The strategy that exploits them is still a page away, but the direction is clear — judge contamination during generation rather than before it.

Page 3 of the paper: Lu: So we've got the metric nailed down and two observations about generation behavior — the greedy collapse on leaked questions, and the runner-up pattern. Page three turns those observations into a design principle and surveys the related work.

Jane: Right, and the critique of existing strategies is structural. TED, LNE-blocking, and shortcut neuron patching all work in two steps — first estimate where the contamination lies, whether that's responses, questions, or neurons, then operate on the estimate. The intervention is only as correct as that estimate. What it misses keeps its inflated performance, and what it wrongly flags suffers unnecessary damage.

Tom: RailCap flips that completely. Instead of a one-shot estimate before decoding, it judges contamination step by step during generation. Every time a sample falls back onto the greedy trajectory, it suppresses the next trajectory token by capping it to the runner-up. Suppression accumulates across steps until the response distribution becomes sufficiently dispersed.

Meng: So the amount of intervention each question gets is decided online by what the sampling actually does, not fixed in advance by a guess. That's the fundamental break from prior work, and it's why the paper describes RailCap as step-wise supervision during generation.

Lu: The contributions list is worth reading closely. Three items — the SA-PPG metric that fixes the two flaws of G-AP and closes the equal-weighting loophole, the RailCap strategy with its online judgment, and the empirical claim that G-AP systematically overestimates the restoration of prior strategies. Each one maps directly to a problem the paper identified earlier.

Lalam: What I find striking is how the related work positions this. The community has detection methods like min-k percent and perplexity that ask whether the model has seen the data, dataset-side work that rebuilds benchmarks, and mitigation that tries to fix the model at decoding time. The paper sits firmly in that third camp but insists the metric has to come first — otherwise you're comparing strategies with an unreliable instrument.

Jane: And notably, the three prior mitigation methods operate at different granularities — TED at the response level, LNE-blocking at the question level, shortcut patching at the neuron level. They had never been compared under a single metric before. This paper does exactly that, and the comparison doesn't flatter them.

Tom: So the stage is set — the metric is defined, the strategy is motivated, and the field landscape is mapped. Next page formalizes the setup and starts building the case that the old readout is too noisy to trust.

Page 4 of the paper: Jane: The landscape is mapped and the design principle is clear. Page four now formalizes the problem — the clean model, the contaminated model, the mitigation strategy — and pins down the first source of error with hard numbers.

Tom: At the center of the setup is a performance readout for each question, and existing work mostly uses a discrete one — sample a single response and mark it one or zero for correctness. The paper's point is that this is essentially one Bernoulli draw from the true solve probability, so a single sample estimates the model's actual performance on that question terribly.

Meng: Wait — two independent batches from the same clean model should look nearly identical.

Lu: That's what makes the reproducibility test so brutal. They draw two independent batches from the same clean model, and under the discrete readout the mean per-question gap between them reaches 0 point 190. Two evaluations of the same model, same questions, and the readout alone differs by almost 0 point 2 — essentially a lottery.

Meng: Under the solve-probability readout, estimated from fifty samples per question, the same two batches collapse onto one curve — the mean per-question gap drops to 0 point 041. So the readout choice isn't cosmetic. It's the difference between noise and signal, and the paper quantifies it directly.

Jane: Then they define G-AP formally — the average readout of the mitigated model minus the average of the clean model, absolute value — and set up the notation for the aggregation problem. Each question contributes a probability gap, and that gap splits into under-suppression, meaning residual contamination left in place, and over-suppression, meaning collateral damage.

Tom: Under G-AP, those two components net against each other, and it's built into the formula. The paper even notes that TED is a partial exception because it works on sampled responses anyway, but its pass@1 estimate is an artifact of its method, not a principled choice.

Lalam: The broader point is that a metric which rewards cancellation will attract strategies that cancel. This page shows the readout alone is already unreliable, and the aggregation formula has cancellation baked in. By the end of the page, both problems are established — and the next page delivers the formal fix.

Page 5 of the paper: Tom: Both problems are now established with numbers — the noisy readout and the cancellation. Page five delivers the formal mathematical fix, and it's surprisingly clean once you see it.

Jane: A-PPG is defined properly — per-question probability gap, absolute value, then average over the dataset. The zero condition is strict: it's zero if and only if every single question's solve probability matches the clean model. Then they build G-APP, which is the same per-question quantity but with the absolute value taken after the average, and the contrast between the two is stark.

Lu: The worked example makes it concrete. Suppose half the questions are over-suppressed by some amount δ and half under-suppressed by the same δ. G-APP reads zero, while A-PPG reads δ. The gap vanishes while not a single question is restored — that's the formal demonstration of the title.

Meng: Then the stratification step handles the equal-weighting problem. The paper partitions questions into fifty equal-width bins according to the clean model's solve probability, averages the per-question gaps within each bin, then averages across bins. The zero-probability majority becomes one group with one vote instead of dominating the whole average.

Tom: And crucially, SA-PPG inherits the strict zero condition from A-PPG. Stratification doesn't relax the per-question requirement — it just stops the trivial fail-everything strategy from looking good. The authors show later that All-Zero, which looks respectable under equal-weight A-PPG, becomes the worst strategy under SA-PPG.

Lu: What I appreciate is that each step of the construction is justified against a specific failure mode. The probability readout fixes reproducibility, per-question differencing fixes cancellation, and stratification fixes the frequency-chasing shortcut. Each one is necessary, and the paper says so explicitly.

Jane: So the measurement side is complete. Any strategy that wants to score well now has to restore questions across the whole difficulty spectrum of the clean model, and that sets the bar for the strategy half of the paper.

Page 6 of the paper: Jane: The ruler is built and it's strict — restoration has to happen at every difficulty level. Page six now introduces RailCap, the strategy designed to meet that bar, and starts setting up the experiments.

Tom: That design principle follows directly from the metric. Because SA-PPG demands per-question accuracy, a strategy needs to know how much to adjust each question, and pre-hoc estimates are fragile. RailCap decides online — the preprocessing is minimal, just one extra greedy decode per question, with an index mapping every trailing n-gram of the trajectory to its successor token.

Lu: Then during sampling, at every decoding step, you check whether the last n tokens match a window of the greedy trajectory. If they do, the trajectory's successor token gets its logit capped to the level of the current runner-up. Otherwise decoding proceeds untouched, and the next token is sampled normally from the adjusted distribution.

Meng: So the memorized path isn't banned — it's flattened to parity with the next-best option. And because the clean model's token is so often that runner-up, it now has a real chance to surface. Suppression accumulates step by step, and the response distribution eventually disperses enough to escape the memorized path entirely.

Tom: The experiments then start with two domains in focus. GSM8K is the standard grade-school math benchmark, and PQ is a paraphrased version they construct where only the wording changes — all numbers and final answers stay identical. PQ matters because verbatim memorization can't directly hit it, making it a harder form of contamination.

Jane: Three model families are used — Llama-2-7B, Gemma-4-E2B, and Pythia-12B. Pythia is the interesting one because its training data is fully public, so they can verify the base model isn't already contaminated by the evaluation data.

Lu: Contamination is simulated in a controlled way — fine-tune on training data to get a clean model, then continue fine-tuning with test questions mixed in. Six hundred sixty leaked questions, six hundred fifty-nine unleaked. Every experiment has known ground truth about which questions leaked, which is what makes the metric comparison meaningful in the first place.

Page 7 of the paper: Tom: We've got the controlled setup — known leaks, three models, two domains. Page seven delivers the headline result, and it's a genuine rank reversal that flips prior conclusions upside down.

Jane: Table one shows the same responses, the same contaminated model, the same clean model, the same strategies — only the metric changes. Under G-AP, LNE-blocking looks near-perfect. Its gap is 0 point 0235, against 0 point 3192 for the contaminated model with no intervention, making it clearly the best strategy in the table.

Lu: So the same method that looked almost perfect suddenly lands near the bottom?

Jane: Exactly — under SA-PPG it collapses to 0 point 2932, barely better than doing nothing at 0 point 3261. And RailCap, which wasn't the best under G-AP, becomes the best under SA-PPG at 0 point 1914. Same everything, different ruler, opposite conclusion.

Meng: The evaluation protocol is standard stuff — fifty samples per question at temperature 0 point 7, solve probability estimated as the fraction correct, fifty stratification bins, eight-shot chain-of-thought prompting. These aren't exotic choices, so the reversal can't be dismissed as an artifact of an unusual setup.

Tom: Page eight's decomposition explains where the old metric buries the error, but the numbers here already tell the story. The near-perfect restoration that prior work claimed simply doesn't exist at the per-question level, and the paper says that in plain terms.

Jane: And this is the first time these three prior strategies — TED, LNE-blocking, shortcut neuron patching — are compared under a single metric. The comparison never happened before, and now that it does, the rankings change fundamentally.

Lu: One more number worth holding onto — the readout noise from earlier. A single 0/1 sample per question gives a mean per-question gap of 0 point 19 between two clean batches, so any metric built on that foundation is measuring noise as much as restoration. The component analysis on the next page shows exactly where that noise hides.

Page 8 of the paper: Lu: The rank reversal is on the table, so the natural question is where the old metric hides its error. Page eight answers that with the component decomposition, and then delivers the full six-setting comparison and the ablations.

Tom: That decomposition splits SA-PPG into under-suppression — residual contamination left untouched — and over-suppression, the collateral damage. LNE-blocking's over-suppression component is 0 point 0836, double RailCap's 0 point 0420, yet its net gap reading is almost four times better because the two components cancel.

Meng: So the old metric rewarded destroying capabilities?

Tom: Exactly — the strategy that broke the most questions looked the best because the breakage cancelled out. The paper calls it out bluntly: the false perfection in the earlier table is not an artifact of estimation noise, it's built into the aggregation.

Jane: Then there's the All-Zero reference — a synthetic strategy that makes the model fail every question. Under the equal-weight A-PPG it scores 0 point 2190, better than the contaminated model itself at 0 point 3793. Under SA-PPG it becomes the worst of all strategies at 0 point 4903. That single comparison is the cleanest proof that stratification is doing real work.

Meng: The main strategy table then covers all six settings — two contamination domains across three models. RailCap wins every single column. Its best is Llama-2 on GSM8K at 0 point 1914, with shortcut patching the runner-up at 0 point 2476. TED is nearly indistinguishable from doing nothing, and LNE-blocking is actually worse than Identity on all three models in the paraphrased PQ domain.

Lu: That domain pattern fits the paper's argument perfectly. In PQ, the questions seen at inference differ from the contaminated ones, so a one-shot estimate made before decoding becomes harder. The estimate-based approaches suffer exactly where their assumption breaks down, while RailCap's online judgment is unaffected.

Tom: The ablations show the design is robust. The n-gram threshold — n equals one triggers too often and causes heavy collateral damage, n equals four balances residual contamination and damage, and everything from three to seven stays within 0 point 008 of the best. The mechanism isn't knife-edged.

Jane: Suppression form matters as well. Capping the trajectory token to the runner-up beats banning it entirely — the hard ban raises SA-PPG from 0 point 1914 to 0 point 2190. Keeping the memorized token available at reduced probability is gentler than prohibiting it outright.

Meng: So the online, step-wise judgment isn't just philosophically different — it's empirically better everywhere, and it degrades gracefully when you change its knobs. That combination is what makes the contribution convincing.

Lalam: And that's the sign of a robust engineering result. It works not because it was tuned to one setup, but because the mechanism itself tracks the generation behavior and adapts question by question. That's why it holds across models and domains — which leaves the question of what this means for the field going forward.

Conclusion: Tom: We've reached the end of the paper, and it's worth stepping back to see the whole arc. The authors took a widely used metric, showed that its zero doesn't mean what everyone assumed, and replaced it with one where zero genuinely certifies per-question restoration.

Jane: The new ruler, SA-PPG, demanded that strategies restore questions across the whole difficulty spectrum, and that exposed a hard truth — prior strategies were far less effective than claimed. LNE-blocking's apparent near-perfection under G-AP turned out to be cancellation between collateral damage and residual contamination.

Lu: RailCap took the design consequences seriously. No estimate of where contamination lies, just real-time observation of whether sampling falls back onto the greedy trajectory, with suppression applied step by step. And it delivered the lowest SA-PPG in all six settings tested.

Meng: The empirical record is clean across three model families and two contamination domains, including paraphrased questions that verbatim memorization can't hit. And the ablations showed the method is robust to its own hyperparameters — that's the mark of a mechanism that's working as intended.

Lalam: Beyond this benchmark, the implication is bigger. A zero gap between averages can be a mirage, and the paper shows it concretely. For the wider community, the lesson is that metrics whose zero means something real are worth the extra effort — otherwise people keep designing toward the mirage.

Tom: And there's a practical warning for anyone reading evaluation scores. If the test data might have leaked into training, a matching average score tells you very little about whether capability was actually restored. The per-question view is the only honest view.

Jane: The paper also leaves open threads — the paraphrased domain was harder for estimate-based methods, which points future work toward contamination that isn't verbatim, and toward metrics that capture even finer-grained restoration behavior.

Lu: There's a reproducibility lesson that extends beyond contamination too. A metric built on single samples can't reproduce itself, so moving to probability-based readouts is a principle that applies to evaluation design generally. That's a valuable side effect of this work.

Tom: Our thanks to the authors — the Zhejiang University team — for giving the community a sharper ruler and a stronger strategy. It's a combination that should change how contamination mitigation gets judged from here on.

Jane: And with that, we close the discussion. A genuinely thought-provoking paper, and we'll be watching for the follow-ups.