Winning by Peeking: Unenforced Budgets and Test-Set Selection Inflate Short-Budget AutoML Comparisons

arXiv:2608.07303 · cs.AI, cs.LG, math.ST, stat.TH · Submitted 2026-08-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Winning by Peeking: Unenforced Budgets and Test-Set Selection Inflate Short-Budget AutoML Comparisons".

Jane: The paper was written by Guilin Zhang and Kai Zhao from Independent Researcher.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title and Authors: Tom: That title gives the whole game away — the system wins by peeking at the test set, and the paper is the confession. We introduced this paper at the top of the show, and I'd say the title alone tells you the shape of the story.

Jane: Right, and the people telling the story are Guilin Zhang and Kai Zhao — independent researchers, no big lab behind them. They built their own AutoML engine, a small tool called Orcetra, and it seemed to beat two serious frameworks, FLAML and AutoGluon, by enormous margins.

Lu: Enormous is the word. And the setting is what makes that relevant, because these comparisons run at 30 or 60 seconds per dataset — the regime you find in README files and blog posts, not the one-to-four-hour budgets that proper benchmark studies use.

Meng: So the short-budget numbers are the ones practitioners actually read when they're choosing a tool.

Tom: Exactly, and if those are inflated, people are making real decisions on wrong information.

Jane: What I find striking is the paper's admission that none of the individual problems is new. Selection bias in model comparison is fifteen years old as a formal result, and the literature on reusing holdout sets goes back years. But nobody had measured the combination in this specific regime.

Meng: So they measured it, on their own system. They present the original result in full, then they dissect what each defect contributed, then they re-run everything with the protocol fixed and release all the code and per-dataset results.

Lalam: Which could change how people work, because the implication is that an ordinary results table — the kind you see in tool announcements every week — can hide this completely. Nothing in the scores tells you the protocol was broken.

Lu: And Orcetra is just 1,661 lines of Python doing guided random search over a fixed pool of scikit-learn models. The authors stress that nothing in the design explains the margins.

Tom: Right, a deliberately boring system with a broken protocol beat two established frameworks. The numbers were so lopsided, so stable across task types and across two budgets, that the result looked beyond dispute. It wasn't.

Summary and Main Findings: Jane: So we're past the setup — let's look at the original protocol's numbers. On 513 OpenML datasets with a nominal 60-second budget, Orcetra won 57 point 1 percent, AutoGluon took 21 point 6, FLAML took 10 point 9, and the rest were ties.

Tom: Against FLAML alone at 30 seconds it won 78 point 4 percent. The head-to-head sign test gives p equals 9 point 5 times ten to the minus forty-six — the kind of significance level that normally ends the conversation.

Lu: But the paper identifies three defects you can't see in any results table. First, the search loop scored every candidate pipeline on the test split and reported the best score it had seen. So the headline was a maximum over dozens of noisy estimates, while the baselines selected on training data and touched the test set exactly once.

Meng: That's the textbook selection bias, with one nasty twist. The number of selection events grows with the budget, so the bias grows with the very thing the experiment is varying — the compute you give the search.

Jane: Second defect — the budget. Orcetra checked the clock between candidates but never interrupted a model fit that was already running. A random forest launched at 59 seconds on a large dataset just runs to completion. So at a nominal 60-second budget, Orcetra consumed a median of 120 seconds, exceeded the budget on 78 percent of datasets, and used 2 point 24 times AutoGluon's actual wall-clock.

Tom: And then the third defect is almost embarrassing in its ordinariness. A later regression-only re-run was sitting in the result directory next to the original sweep. Deduplicating every result file by dataset ID — the obvious move — silently merged the two runs and lifted the headline from 57 point 1 to 61 point 2 percent.

Lu: What changed in that re-run was the competitors, not Orcetra. Orcetra's score was bit-identical on 117 of 131 datasets, while AutoGluon scored worse on 88 of them. The machine was more heavily loaded, so the frameworks that respect wall-clock did fewer trials, and the one that doesn't simply took longer.

Meng: So they re-ran with the corrected protocol — selection on a validation split, the test set scored exactly once, the deadline enforced from outside the search, and every framework pinned to an equal share of the machine.

Jane: On that re-run subset, Orcetra's win rate collapses from 59 point 4 percent to 34 point 3. FLAML rises to 28 point 0, AutoGluon to 27 point 3, and no pairwise difference against either competitor remains significant. The original margin was an artifact.

Tom: But here's the surprise — when they separate the two corrections, the test-set selection is worth only 4 point 8 percentage points. Most of the collapse came from the compute inequality, and I want to talk about how they could measure that separation.

Improvements and Recommendations: Jane: The measurement trick is the part I find genuinely clever. Instead of re-running and comparing aggregate win rates, which would confound the protocol change with run-to-run variance, they record both estimands inside a single search. Every candidate is fitted once and predicted twice — on validation, which drives selection, and on test, which drives nothing.

Tom: So the old number and the new number come from the same candidates, the same training data, the same budget. The only difference is the selection rule. Measured that way, selecting on the test set buys 4 point 8 percentage points — real, but nowhere near the twenty-five points that separate the original and corrected headlines.

Lu: The rest of the damage belongs to compute. FLAML is the main beneficiary of a fair machine, going from 11 point 2 percent to 28 point 0 percent on the same datasets, because under contention it kept its wall-clock and gave up trials while Orcetra kept its trials and gave up wall-clock.

Meng: The traces also let them measure the selection bias directly as a function of budget. They log every candidate's test score, so at any budget the realized bias is the difference between the best test score and what the validation split would have picked. It grows with the number of candidates, as the theory says, but it reaches only about 0 point 27 accuracy points.

Tom: That's roughly five times smaller than the closed-form bound they started with — the square root of two log K times the standard error, which predicted around 1 point 3 points. The reason it overestimates is that all candidates are scored on the same test rows, so the common noise cancels out of a maximum.

Jane: So they keep the bound but demote it to a screen. If your margin is smaller than the bound, you can't trust it. It's just not a predictor of the actual bias.

Lu: Then comes the checklist, and this is the part I want every tool author to read. Report realized wall-clock, not just the nominal budget. Report how many candidates the search evaluated. Trace the evaluation data through the search code and count how often the test set influences a decision — that number has to be one.

Meng: And enforce the deadline externally, in a killable process. Treat the tie threshold as a claim about precision — declaring ties at ten to the minus six on a metric whose standard error is ten to the minus two means near-ties get resolved by noise.

Tom: What makes the list practical is how cheap it is. The first two items are logging; the rest is ten lines of code and a habit. None of it requires a containerized benchmark harness.

Jane: And that's the encouraging ending — these fixes are affordable. I want to go back to the opening pages now, because the way the paper frames the problem sets up everything we've been discussing.

The First Page: Tom: The opening pages set the stakes pretty sharply. It says the benchmarks practitioners trust run at one to four hours per dataset, but the comparisons practitioners actually read run at 30 or 60 seconds — the tool READMEs, the blog posts, the workshop submissions.

Jane: And that's the only regime where a solo developer can afford to sweep several hundred datasets. So the paper argues this is where protocol errors do the most damage and are least likely to be caught.

Lu: The abstract also flags the origin story, because Orcetra started as a forecasting system for Polymarket. The paper includes a calibration study on 178 resolved contracts, and it leans toward the classic favorite-longshot bias — low-probability contracts trading above their realized frequency — but it doesn't reach significance once you pool the sample.

Meng: They're scrupulous about that. The most striking bin on the reliability diagram is the 0 point 20-to-0 point 30 band, where the market implied 25 percent and only 12 percent resolved true. But that's the minimum over ten bins, and its standard error is large enough that after accounting for looking at ten bins, it's not evidence. The pooled test gives z equals 1 point 7, p equals 0 point 087 — right direction, not significant.

Tom: Including that case study takes nerve, because it's an instance of the paper's own thesis in a second domain. Pick the best-looking bin and you have a story. Pool the data and the story weakens.

Jane: And the first page finishes by conceding that none of the individual observations is novel. Selection bias in model comparison is fifteen years old in exactly this form, and the adaptive reuse of a holdout has a whole literature behind it.

Lalam: But the combination is what's worth reporting — how large the distortion gets in the exact regime where informal comparisons happen, and how completely a conventional results table hides it. The paper describes the whole thing as a measurement rather than a new theorem.

Tom: One line from the paper stays with me — this kind of defect doesn't produce implausible numbers. It produces a plausible one-to-two-point edge, sustained across hundreds of datasets, with the tiny margins that a real but modest improvement would also produce. That's why it survives in the wild, and it's why the conclusion here deserves attention.

Conclusion: Lalam: So let's take stock of the whole arc. A 1,661-line random search over a fixed model pool appeared to beat FLAML and AutoGluon on 513 OpenML datasets, with margins that were consistent across task types, stable across two budgets, and significant at levels like ten to the minus forty-six. That's the kind of evidence that normally settles a debate.

Jane: And the paper shows the victory was largely an illusion. Orcetra reported the best of dozens of test-set evaluations while its competitors reported one, and it took a median 2 point 24 times AutoGluon's wall-clock under the same nominal budget. Fix the protocol and the win rate falls from 59 point 4 to 34 point 3 percent, with no pairwise difference left significant.

Tom: The part I'll remember is the paired measurement design — recording both estimands inside one run, at the cost of one extra prediction per candidate. That's what let them say the selection rule was worth 4 point 8 points and unequal compute was worth most of the rest.

Lu: And the measured selection curve corrects the theory in an interesting way. The bias does grow with the budget, but it reaches about 0 point 27 accuracy points instead of the 1 point 3-point worst-case bound, because candidates scored on shared rows cancel most of their noise. The bound is a screen, not a prediction.

Meng: The checklist distills the whole thing into habits anyone can adopt — report realized wall-clock, log candidate counts, trace what the search reads, compare margins to the noise floor, enforce deadlines externally, and treat tie thresholds as claims about precision.

Lalam: There's a broader point about where evaluation error concentrates. Careful benchmarks run for hours on cross-validated folds, where a one-point margin means something. The comparisons that circulate run for seconds on single splits, where the noise floor is high, the number of selection events is large, and the budget connects the two.

Jane: And that's why this paper matters beyond AutoML. Any field comparing methods on quick, informal benchmarks carries the same risk.

Tom: Agreed. We'll say goodbye to this paper and get ready for the next one.

Guilin Zhang, Kai Zhao

Independent Researcher

cs.AI, cs.LG, math.ST, stat.TH

Submitted: 2026-08-07

Updated: 2026-08-10

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 69/100

The gist: This paper is a case study in how informal, short-budget AutoML comparisons can produce dramatically inflated results due to protocol defects that are invisible in a conventional results table.

Key concepts

Selection bias: When a model is chosen based on its performance on a test set, the reported score is an overestimate because it's the maximum of many noisy evaluations. The paper shows this bias grows with the number of candidates evaluated, inflating results in short-budget comparisons.
Wall-clock budget: The actual time a system takes to run, as opposed to a nominal budget. Orcetra exceeded its 60-second budget by not interrupting running models, using 2.24 times AutoGluon's real time, giving it an unfair advantage in trials.
Paired measurement design: A method to isolate the effect of a protocol change by recording both old and new metrics within the same run. Here, each candidate was scored on validation and test sets, allowing the authors to attribute the win-rate drop to test-set selection (4.8 points) versus compute inequality.

Terminology

Accuracy — 22× in this paper · explained in 1 episode
Benchmark — 22× in this paper · explained in 2 episodes
Selection Bias — 12× in this paper · explained in 4 episodes
Calibration — 10× in this paper · explained in 7 episodes
Trace — 9× in this paper · explained in 2 episodes
Core — 8× in this paper · explained in 2 episodes
Benchmarking — 3× in this paper · explained in 4 episodes
Meta-learning — 2× in this paper · explained in 2 episodes
Feature Engineering — 1× in this paper · explained in 3 episodes
Mobile Edge Computing — 1× in this paper · explained in 2 episodes
Pairwise Comparisons — 1× in this paper · explained in 2 episodes
Robustness — 1× in this paper · explained in 5 episodes
Uncertainty Quantification — 1× in this paper · explained in 8 episodes
Verification — 1× in this paper · explained in 2 episodes

Summary

This paper is a case study in how informal, short-budget AutoML comparisons can produce dramatically inflated results due to protocol defects that are invisible in a conventional results table. The authors built their own simple AutoML engine, Orcetra — a 1,661-line Python random search over a fixed scikit-learn model pool — and initially found that it appeared to beat FLAML and AutoGluon on 513 OpenML datasets, winning 57.1% of them at a nominal 60-second budget and 78.4% of datasets against FLAML alone at 30 seconds. The authors then discovered that Both margins came from protocol defects that a results table cannot show.

Selection on the test set. "Orcetra's search loop scored each candidate pipeline by fitting it on the training split and evaluating it on the test split, then reported the best score it had seen. Over a 60-second budget the loop evaluates a median of several dozen candidates, so the reported number is a maximum over dozens of noisy estimates of the same quantity, each computed on the same held-out sample. FLAML and AutoGluon select internally by cross-validation on training data and touch the test set exactly once. The reported score is therefore the maximum over K noisy test evaluations: the comparison therefore contrasts an oracle-selected score against two honestly held-out scores."
Budgets checked but not enforced. "The loop tested elapsed < budget before starting a candidate and never interrupted one in flight. As a result, at a nominal 60-second budget, Orcetra consumed a median of 120 s per dataset against 70 s for FLAML and 62 s for AutoGluon, exceeding its own budget on 78% of datasets and doubling it on 50%." The median per-dataset wall-clock ratio was 1.46× versus FLAML and 2.24× versus AutoGluon.
Results waiting to be spliced. A later regression-only re-run sat in the result directory beside the original sweep with nothing marking which supersedes which. Deduplicating every result file by dataset ID — the obvious move, and the one our own first analysis made — merges them and lifts the headline to 61.2%. The re-run was supposed to measure a new ensembling step, but the ensemble never wins, and Orcetra's own scores are unchanged on 117 of 131 datasets. What moved was the competitors, which scored worse under the heavier machine load of the later run.

The authors re-ran the three-way comparison with both main defects fixed: selection moved to a validation split carved from the training data, and the budget was enforced externally by running the search in a killable child process with a grace period. Every framework got an equal pinned share of the machine (four workers, each pinned to five cores).

Using a paired-measurement design — each candidate fitted once but scored on both validation and test, recording both estimands inside a single run — they measured the effect of each defect in isolation. Over 143 re-run datasets (105 classification, 38 regression):

Under the original protocol, Orcetra won 59.4% of datasets.
Under the corrected protocol, Orcetra's win rate falls to 34.3%, with FLAML at 28.0% and AutoGluon at 27.3%, and 10.5% ties.
Head to head: it now beats FLAML 73–62 (p = 3.9 × 10−1) and AutoGluon 68–62 (p = 6.6 × 10−1). Neither is distinguishable from a coin flip.

On the attribution: the selection rule accounts for 4.8 percentage points and unequal compute for most of the rest. FLAML went from 11.2% to 28.0% on the same datasets — the system most starved by the original contention regime... Unequal compute, not peeking, did most of the damage here.

The paper measures the realized selection bias as a function of budget rather than assuming it: "the bias does grow with K, but reaches only 0.27 accuracy points — about five times smaller than the sigma√(2 ln K) bound that a marginal-standard-error argument predicts, because candidates scored on shared test rows cancel most of the noise. The closed-form screen from §6.1 predicted 1.30 accuracy points of inflation as an upper bound; the measured realized bias at the largest budget was 0.27 points. The bias is concentrated, not diffuse": "across the 79 paired classification datasets the gap is exactly zero on two thirds of them... On the remaining third it buys a great deal — the distribution has mean 0.27 accuracy points, upper quartile 0.11, and a maximum of 3.25. This concentration matters because a win rate is decided by the datasets where the margin is thin, and those are the datasets where peeking pays."

Compute-matched slicing of the original run: restricting to datasets where Orcetra used no more wall-clock than FLAML drops its win rate from 57.1% to 49.4%; requiring it used no more than AutoGluon drops it to 43.1–44.4%.
A prediction-market calibration case study (§8): on 178 resolved Polymarket contracts, the framework recovers something leaning toward the favourite–longshot bias but not significant at conventional levels (pooled z = 1.7, p = 0.087 two-sided). The authors emphasize that quoting the best single bin would be selecting a maximum and reporting it as a measurement — the same mistake the paper is about.
A checklist for short-budget comparisons (§9): report realized wall-clock, not the budget passed; report the number of candidates evaluated; trace the evaluation data through the search code; compare margins against the noise floor before believing them; prefer paired same-run measurements; enforce budgets externally; treat tie thresholds as claims about precision.

The paper concludes: "A 1,661-line random search over a fixed scikit-learn model pool appeared to beat FLAML and AutoGluon on 513 OpenML datasets, by margins that were consistent across task types, stable across two budgets, and significant at p = 9.5 × 10−46. It did so because it reported the best of dozens of test-set evaluations while its competitors reported one, and because it took a median 2.24× the per-dataset wall-clock AutoGluon did under the same nominal budget. Correcting the protocol takes it from 59.4% to 34.3% on the re-run subset, where no pairwise difference against either competitor is significant."

Improvements for AI systems

Based on the paper, here are the specific improvements for AI systems, particularly AutoML engines and benchmark harnesses:

Improvement: Change the search loop so candidate pipelines are scored only on a validation split carved from training data; the test set is touched exactly once, after the final model is selected.
What the improved system can do: Report honestly held-out performance instead of the maximum over dozens of noisy test evaluations. This removes the “oracle-selected score” inflation that made a 1,661-line random search appear to beat FLAML and AutoGluon by 57–78% of datasets.
Improvement: Run the search in a killable child process with a grace period, and measure actual wall-clock time from outside the process. Do not rely on the search loop checking elapsed < budget between candidates, which lets long-running candidates overrun the budget.
What the improved system can do: Guarantee that reported results correspond to the nominal budget. The paper’s system consumed a median 120 s at a 60 s budget (2.24× AutoGluon’s actual time); an externally enforced system will not silently get 2× more compute than competitors.
Improvement: Log the actual elapsed time and the number of models evaluated for every run, and make these fields mandatory in results tables.
What the improved system can do: Let users and benchmarks distinguish “budget passed” from “compute actually used.” Compute-matched slicing then becomes possible—restricting comparisons to datasets where the system used no more wall-clock than competitors dropped its win rate from 57.1% to 49.4% against FLAML and 43–44% against AutoGluon.
Improvement: Fit each candidate once, then score it on both validation and test inside the same run, recording both estimands. This isolates protocol effects from machine-load and dataset drift.
What the improved system can do: Quantify the bias of selection-on-test-set directly (the paper measured 0.27 accuracy points realized, vs. a 1.30-point theoretical upper bound) and attribute win-rate changes precisely—showing that unequal compute, not peeking, did most of the damage.
Improvement: Pin each framework to an equal number of cores/workers and run competitors under identical machine-load conditions, rather than letting one framework grab more resources or run while others contend.
What the improved system can do: Produce fair head-to-head comparisons. In the corrected protocol, Orcetra fell to 34.3% wins with FLAML at 28.0% and AutoGluon at 27.3%; neither pairwise difference was significant, and the previously starved system (FLAML) went from 11.2% to 28.0% win rate.
Improvement: Define ties using a meaningful precision threshold (e.g., 1e-4 accuracy), report margin distributions, and compare margins against the noise floor before claiming a win.
What the improved system can do: Avoid declaring victory on datasets where the margin is thinner than the noise—exactly where peeking pays. The paper found the selection bias was concentrated: zero on two-thirds of datasets but up to 3.25 accuracy points on the remaining third.
Improvement: Mark result directories with metadata (timestamp, protocol version, config hash) and never merge results by dataset ID alone without checking which run supersedes which.
What the improved system can do: Prevent “results waiting to be spliced” from corrupting a benchmark—a regression-only re-run mixed with the original sweep lifted the headline from 57.1% to 61.2% purely by making competitors look worse under heavier load, not by improving the system.
Improvement: Add assertions/tracing in the search loop that verify which split is used for scoring, and raise an error if the test split appears anywhere in the candidate-selection path.
What the improved system can do: Make protocol defects visible in the codebase rather than invisible in a results table, so future short-budget comparisons—and LLM-driven AutoML agents—cannot silently acquire an oracle.
Improvement: When LLM-based agents perform search over prompts, tools, or reasoning strategies, require validation-set selection before any test-set or benchmark evaluation, and enforce killable per-attempt budgets.
What the improved system can do: Report benchmark numbers that reflect genuine generalization rather than the best of K test-set evaluations, preventing the same “selection-on-test” inflation that made a random search over a fixed model pool look like a state-of-the-art AutoML system.

Abstract

Comparisons between AutoML systems at short time budgets -- tens of seconds rather than hours -- are common in tool READMEs and workshop papers, and they are easy to get wrong. We report a case study in which a simple AutoML engine, Orcetra, appeared to beat FLAML and AutoGluon on 513 OpenML datasets, winning 57.1% of them at a nominal 60-second budget and 78.4% of datasets against FLAML alone at 30 seconds. Both margins came from protocol defects that a results table cannot show. The search loop scored every candidate on the test split and reported the best, making the headline metric a maximum over dozens of noisy estimates while the baselines selected on training data and touched the test set once; and the budget was checked before launching a candidate but never enforced during one, so the system consumed a median of 120 s against a 60-second budget, 2.24x the wall-clock AutoGluon used. Re-running with selection moved to a validation split, the deadline enforced externally and every framework pinned to an equal share of the machine, Orcetra's win rate on the re-run subset falls from 59.4% to 34.3% and no pairwise difference against either competitor remains significant. Recording both estimands inside a single search lets us attribute the collapse: the selection rule accounts for 4.8 percentage points and unequal compute for most of the rest. The same traces give the selection bias as a function of budget, measured rather than assumed: it grows with K but reaches only 0.27 accuracy points, about five times below the sigma sqrt 2 K bound a marginal-standard-error argument predicts, because candidates scored on shared test rows cancel most of the noise. We close with a checklist for short-budget comparisons. Code, per-dataset results and the scripts that regenerate every number and figure in the paper are released with it.