2608.07303-Winning by Peeking: Unenforced Budgets and Test-Set Selection Inflate Short-Budget AutoML Comparisons

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Winning by Peeking: Unenforced Budgets and Test-Set Selection Inflate Short-Budget AutoML Comparisons".

Jane: The paper was written by Guilin Zhang and Kai Zhao from Independent Researcher.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title and Authors: Tom: That title gives the whole game away — the system wins by peeking at the test set, and the paper is the confession. We introduced this paper at the top of the show, and I'd say the title alone tells you the shape of the story.

Jane: Right, and the people telling the story are Guilin Zhang and Kai Zhao — independent researchers, no big lab behind them. They built their own AutoML engine, a small tool called Orcetra, and it seemed to beat two serious frameworks, FLAML and AutoGluon, by enormous margins.

Lu: Enormous is the word. And the setting is what makes that relevant, because these comparisons run at 30 or 60 seconds per dataset — the regime you find in README files and blog posts, not the one-to-four-hour budgets that proper benchmark studies use.

Meng: So the short-budget numbers are the ones practitioners actually read when they're choosing a tool.

Tom: Exactly, and if those are inflated, people are making real decisions on wrong information.

Jane: What I find striking is the paper's admission that none of the individual problems is new. Selection bias in model comparison is fifteen years old as a formal result, and the literature on reusing holdout sets goes back years. But nobody had measured the combination in this specific regime.

Meng: So they measured it, on their own system. They present the original result in full, then they dissect what each defect contributed, then they re-run everything with the protocol fixed and release all the code and per-dataset results.

Lalam: Which could change how people work, because the implication is that an ordinary results table — the kind you see in tool announcements every week — can hide this completely. Nothing in the scores tells you the protocol was broken.

Lu: And Orcetra is just 1,661 lines of Python doing guided random search over a fixed pool of scikit-learn models. The authors stress that nothing in the design explains the margins.

Tom: Right, a deliberately boring system with a broken protocol beat two established frameworks. The numbers were so lopsided, so stable across task types and across two budgets, that the result looked beyond dispute. It wasn't.

Summary and Main Findings: Jane: So we're past the setup — let's look at the original protocol's numbers. On 513 OpenML datasets with a nominal 60-second budget, Orcetra won 57 point 1 percent, AutoGluon took 21 point 6, FLAML took 10 point 9, and the rest were ties.

Tom: Against FLAML alone at 30 seconds it won 78 point 4 percent. The head-to-head sign test gives p equals 9 point 5 times ten to the minus forty-six — the kind of significance level that normally ends the conversation.

Lu: But the paper identifies three defects you can't see in any results table. First, the search loop scored every candidate pipeline on the test split and reported the best score it had seen. So the headline was a maximum over dozens of noisy estimates, while the baselines selected on training data and touched the test set exactly once.

Meng: That's the textbook selection bias, with one nasty twist. The number of selection events grows with the budget, so the bias grows with the very thing the experiment is varying — the compute you give the search.

Jane: Second defect — the budget. Orcetra checked the clock between candidates but never interrupted a model fit that was already running. A random forest launched at 59 seconds on a large dataset just runs to completion. So at a nominal 60-second budget, Orcetra consumed a median of 120 seconds, exceeded the budget on 78 percent of datasets, and used 2 point 24 times AutoGluon's actual wall-clock.

Tom: And then the third defect is almost embarrassing in its ordinariness. A later regression-only re-run was sitting in the result directory next to the original sweep. Deduplicating every result file by dataset ID — the obvious move — silently merged the two runs and lifted the headline from 57 point 1 to 61 point 2 percent.

Lu: What changed in that re-run was the competitors, not Orcetra. Orcetra's score was bit-identical on 117 of 131 datasets, while AutoGluon scored worse on 88 of them. The machine was more heavily loaded, so the frameworks that respect wall-clock did fewer trials, and the one that doesn't simply took longer.

Meng: So they re-ran with the corrected protocol — selection on a validation split, the test set scored exactly once, the deadline enforced from outside the search, and every framework pinned to an equal share of the machine.

Jane: On that re-run subset, Orcetra's win rate collapses from 59 point 4 percent to 34 point 3. FLAML rises to 28 point 0, AutoGluon to 27 point 3, and no pairwise difference against either competitor remains significant. The original margin was an artifact.

Tom: But here's the surprise — when they separate the two corrections, the test-set selection is worth only 4 point 8 percentage points. Most of the collapse came from the compute inequality, and I want to talk about how they could measure that separation.

Improvements and Recommendations: Jane: The measurement trick is the part I find genuinely clever. Instead of re-running and comparing aggregate win rates, which would confound the protocol change with run-to-run variance, they record both estimands inside a single search. Every candidate is fitted once and predicted twice — on validation, which drives selection, and on test, which drives nothing.

Tom: So the old number and the new number come from the same candidates, the same training data, the same budget. The only difference is the selection rule. Measured that way, selecting on the test set buys 4 point 8 percentage points — real, but nowhere near the twenty-five points that separate the original and corrected headlines.

Lu: The rest of the damage belongs to compute. FLAML is the main beneficiary of a fair machine, going from 11 point 2 percent to 28 point 0 percent on the same datasets, because under contention it kept its wall-clock and gave up trials while Orcetra kept its trials and gave up wall-clock.

Meng: The traces also let them measure the selection bias directly as a function of budget. They log every candidate's test score, so at any budget the realized bias is the difference between the best test score and what the validation split would have picked. It grows with the number of candidates, as the theory says, but it reaches only about 0 point 27 accuracy points.

Tom: That's roughly five times smaller than the closed-form bound they started with — the square root of two log K times the standard error, which predicted around 1 point 3 points. The reason it overestimates is that all candidates are scored on the same test rows, so the common noise cancels out of a maximum.

Jane: So they keep the bound but demote it to a screen. If your margin is smaller than the bound, you can't trust it. It's just not a predictor of the actual bias.

Lu: Then comes the checklist, and this is the part I want every tool author to read. Report realized wall-clock, not just the nominal budget. Report how many candidates the search evaluated. Trace the evaluation data through the search code and count how often the test set influences a decision — that number has to be one.

Meng: And enforce the deadline externally, in a killable process. Treat the tie threshold as a claim about precision — declaring ties at ten to the minus six on a metric whose standard error is ten to the minus two means near-ties get resolved by noise.

Tom: What makes the list practical is how cheap it is. The first two items are logging; the rest is ten lines of code and a habit. None of it requires a containerized benchmark harness.

Jane: And that's the encouraging ending — these fixes are affordable. I want to go back to the opening pages now, because the way the paper frames the problem sets up everything we've been discussing.

The First Page: Tom: The opening pages set the stakes pretty sharply. It says the benchmarks practitioners trust run at one to four hours per dataset, but the comparisons practitioners actually read run at 30 or 60 seconds — the tool READMEs, the blog posts, the workshop submissions.

Jane: And that's the only regime where a solo developer can afford to sweep several hundred datasets. So the paper argues this is where protocol errors do the most damage and are least likely to be caught.

Lu: The abstract also flags the origin story, because Orcetra started as a forecasting system for Polymarket. The paper includes a calibration study on 178 resolved contracts, and it leans toward the classic favorite-longshot bias — low-probability contracts trading above their realized frequency — but it doesn't reach significance once you pool the sample.

Meng: They're scrupulous about that. The most striking bin on the reliability diagram is the 0 point 20-to-0 point 30 band, where the market implied 25 percent and only 12 percent resolved true. But that's the minimum over ten bins, and its standard error is large enough that after accounting for looking at ten bins, it's not evidence. The pooled test gives z equals 1 point 7, p equals 0 point 087 — right direction, not significant.

Tom: Including that case study takes nerve, because it's an instance of the paper's own thesis in a second domain. Pick the best-looking bin and you have a story. Pool the data and the story weakens.

Jane: And the first page finishes by conceding that none of the individual observations is novel. Selection bias in model comparison is fifteen years old in exactly this form, and the adaptive reuse of a holdout has a whole literature behind it.

Lalam: But the combination is what's worth reporting — how large the distortion gets in the exact regime where informal comparisons happen, and how completely a conventional results table hides it. The paper describes the whole thing as a measurement rather than a new theorem.

Tom: One line from the paper stays with me — this kind of defect doesn't produce implausible numbers. It produces a plausible one-to-two-point edge, sustained across hundreds of datasets, with the tiny margins that a real but modest improvement would also produce. That's why it survives in the wild, and it's why the conclusion here deserves attention.

Conclusion: Lalam: So let's take stock of the whole arc. A 1,661-line random search over a fixed model pool appeared to beat FLAML and AutoGluon on 513 OpenML datasets, with margins that were consistent across task types, stable across two budgets, and significant at levels like ten to the minus forty-six. That's the kind of evidence that normally settles a debate.

Jane: And the paper shows the victory was largely an illusion. Orcetra reported the best of dozens of test-set evaluations while its competitors reported one, and it took a median 2 point 24 times AutoGluon's wall-clock under the same nominal budget. Fix the protocol and the win rate falls from 59 point 4 to 34 point 3 percent, with no pairwise difference left significant.

Tom: The part I'll remember is the paired measurement design — recording both estimands inside one run, at the cost of one extra prediction per candidate. That's what let them say the selection rule was worth 4 point 8 points and unequal compute was worth most of the rest.

Lu: And the measured selection curve corrects the theory in an interesting way. The bias does grow with the budget, but it reaches about 0 point 27 accuracy points instead of the 1 point 3-point worst-case bound, because candidates scored on shared rows cancel most of their noise. The bound is a screen, not a prediction.

Meng: The checklist distills the whole thing into habits anyone can adopt — report realized wall-clock, log candidate counts, trace what the search reads, compare margins to the noise floor, enforce deadlines externally, and treat tie thresholds as claims about precision.

Lalam: There's a broader point about where evaluation error concentrates. Careful benchmarks run for hours on cross-validated folds, where a one-point margin means something. The comparisons that circulate run for seconds on single splits, where the noise floor is high, the number of selection events is large, and the budget connects the two.

Jane: And that's why this paper matters beyond AutoML. Any field comparing methods on quick, informal benchmarks carries the same risk.

Tom: Agreed. We'll say goodbye to this paper and get ready for the next one.