2608.07437-Fisher-R1: Training LLM Agents for Reliable Hypothesis Testing

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Fisher-R1: Training LLM Agents for Reliable Hypothesis Testing".

Jane: The paper was written by Jiacheng Miao, Jin Mu, Guanhua Chen and James Zou from Stanford University and University of Wisconsin–Madison.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary: Tom: Welcome back, everyone. Today we're looking at a paper that tackles something that sounds dry but is actually urgent — whether eye agents can be trusted to run a statistical hypothesis test and get the conclusion right. The authors built a new benchmark of real scientific tasks, then trained their own open-weight model to pass it.

Jane: And the reason it's urgent is that these coding agents are already being used to inspect datasets, write analysis code, and produce reports end to end. If they make subtle inferential mistakes, the fluently written conclusion can still be completely wrong. The paper shows exactly that happening with a frontier model.

Lu: What struck me is the specific failure mode. The model doesn't write broken code. The code runs, produces a precise p-value, and the conclusion still ends up wrong — because the chosen test doesn't fit the data. And existing benchmarks don't catch it, because they rarely check whether the analysis is statistically valid, only whether it executes.

Meng: And the example they open with is perfect. A cancer genomics dataset where a few outliers make linear regression look highly significant, but a rank-based test correctly fails to reject the null. GPT-5 point 4 literally noticed the outliers and the warning signs, then ran the linear regression anyway.

Lalam: So the paper does two main things. It builds P-Bench, 425 real hypothesis-testing tasks with expert-verified answer keys spanning economics, biology, and medicine. And it trains Fisher-R1, an open-weight agent that beats GPT-5 point 4 and DeepSeek-V4-Pro on the strictest scoring. A 14-billion-parameter model doing that tells you something about where the capability gap actually is.

Tom: And that gap lives in statistical judgment — knowing which test is valid given the assumptions of the data. The paper says the method must match the question and the data, otherwise a precise p-value can support the wrong scientific conclusion. That's the thread we'll pull through the whole episode.

Lu: Another thing about the framing — they're not claiming agents can replace statisticians. They're showing the current generation isn't reliable enough even for well-defined single tests. That's a lower bar than people assume.

Jane: It's a good thread. The opening pages lay out the failure mode in detail. And the numbers they cite frame everything else.

Page 1 of the paper: Jane: So the opening section carries one core claim — LLM agents frequently make subtle inferential errors that lead to incorrect conclusions, even when the executed analysis is technically correct.

Tom: Even when the code runs?

Jane: Exactly. The code runs, the p-value is precise, and the conclusion is still wrong, because the test itself wasn't valid for the data. And existing benchmarks fail to capture this, because they rarely assess whether a reported p-value is statistically valid given the assumptions underlying the data.

Lu: And they quantify it with the headline numbers. On P-Bench, Fisher-R1-14B shows a 21 percent average relative improvement in single-trial success over DeepSeek-V4-Pro. And on the most challenging tasks, that gain goes up to 26 percent.

Meng: But the more telling number is the baseline failure rate. Frontier models need that much improvement because they start from a low bar on strict scoring. And the hypothesis-testing workflow they describe is familiar to any scientist — take a question and a dataset, translate it into a testable hypothesis, select a test, compute a p-value, draw a conclusion.

Jane: I love that they show the actual trace in Figure 1. GPT-5 point 4 sees that tumor purity reaches a value of 2 point 47, which should be impossible, sees the linear model looking strong while the rank-based one looks weak, and still reports the linear association as significant.

Tom: Wait — the dataset had a tumor purity above 2 point 5?

Jane: Yes, an outlier that extreme should have been a red flag. Fisher-R1 instead runs the Spearman test, gets p = 0 point 085, and correctly fails to reject the null. The frontier model produces a false discovery; the trained 7-billion-parameter model gets it right.

Lalam: That example carries the whole motivation. It shows why the benchmark and the training pipeline exist, and why conclusion-only evaluation was overstating how reliable these agents are. The p-value has to be statistically valid, not just computable.

Tom: So the next thing they define is exactly that contract — what the agent must deliver and what the hidden answer key checks.

Page 2 of the paper: Jane: So we've seen the failure mode and the scale of it. Now the paper defines the problem formally. The agent receives a scientific question, a dataset, and a data description — but no prescribed method. It has to choose the statistical test, execute the analysis in an R environment, report a p-value, and return a reject or fail-to-reject decision at a pre-specified significance level.

Lu: The multi-turn loop matters too. The agent writes code, sees the output, gets warnings and diagnostics, and can revise. It's not a one-shot answer; it's an iterative process with real feedback, which mirrors how a human analyst works.

Meng: One detail worth keeping — the setting is open-ended by design. The task doesn't tell the agent which test to run. The agent must autonomously select an analysis strategy, and then the output is judged against a hidden answer key that the agent never sees.

Jane: And the related work section makes the landscape clear. Factoid benchmarks ask for a number. Workflow benchmarks check whether code executes and the final answer matches. StatQA handles method selection, but in multiple-choice format without running the analysis. Scientific claim verification checks claims against abstracts, but never reconstructs the underlying statistics.

Lalam: P-Bench sits in the gap, because the answer key isn't transcribed from what a paper claims. It's computed from a logged execution of the canonical reference analysis on the real dataset, then audited by domain experts. That grounding is what makes these tasks usable for both evaluation and training.

Tom: And you need that trust, because the benchmark is the measuring stick for everything else. Which brings up the pipeline question — how do you build 425 verified tasks without an army of human annotators?

Lu: Yeah, that's the construction pipeline. And that's exactly what comes next.

Page 3 of the paper: Tom: So now the construction pipeline. P-Bench draws from three source families — economics papers with datasets on Harvard Dataverse, biology papers with data on cBioPortal, and authoritative biostatistics teaching materials from Vanderbilt. Every task is anchored to an analysis that domain experts already computed and acted on.

Jane: Then the three-stage pipeline. Reproduce the reference analysis on a clean machine and log the execution. Filter out anything that can't be reproduced. Package a self-contained task with a structured answer key in the P-Bench format. So the p-value in the key comes from a logged run of canonical code, not from reading the paper's prose.

Lu: The expert audit closes the loop. Statisticians independently verify that the analysis request, the released data subset, and the answer key all align with the original source. Tasks that fail the review get repaired or removed.

Lalam: And one detail I appreciate — the benchmark deliberately includes realistic statistical traps. Outliers, heteroskedasticity, clustered observations are all in there. So the benchmark checks whether the model computes a number correctly, and it also checks whether the model notices when the data are trying to mislead it.

Meng: The composition numbers tell the same story. 425 tasks total, 203 easy and 222 hard. Hard tasks either belong to tricky method families — Cox regression, instrumental variables, Tobit — or they carry adversarial data-quality perturbations. And the method coverage spans 17 categories, with no single category exceeding 19 percent.

Jane: Those perturbations really matter. Textbook assumption violations that any trained statistician would check for — and the benchmark shows agents walking straight into them. On P-Hard, GPT-5 point 4's strict accuracy drops to 30 point 5 from 64 point 7 on easy.

Tom: That easy-to-hard drop is the measurement gap the field needed. So the benchmark exists and the failure is quantified. But 425 tasks are nowhere near enough to train an agent from scratch — that's where the synthetic data generator comes in.

Page 4 of the paper: Tom: So the synthetic task generator is the key to scaling. It's a Cartesian grid over six factors — statistical method, domain scenario, sample size, effect size, prompt style, and seed. The full corpus comes to 8,642 tasks with balanced coverage across all of them.

Jane: The clever part is the answer key. An LLM writes simulation code that generates a dataset, and then the canonical statistical method runs on its own simulated data. The p-value and the reject or fail-to-reject decision that come out become the ground truth. So the reward signal is verified by construction, not by human labeling.

Lu: And the effect-size axis has three regimes — null, borderline, medium. The borderline regime is auto-calibrated by simulation, so tasks land in a genuinely ambiguous significance range. That's where statistical judgment actually gets tested.

Meng: The data-quality perturbations carry over into training too. Missing values, extreme observations, invalid entries — the model has to notice and handle them rather than mechanically applying a test. That mirrors the traps in P-Bench, so both sets stress the same skills.

Jane: Then the SFT stage. They collect expert trajectories from Claude Sonnet 4 point 6 following a fixed five-step workflow — basic exploration, detailed exploration, assumption checking, method selection and analysis, and the conclusion. Automatic quality control keeps only trajectories with a valid multi-turn trace, a parseable conclusion, and agreement with the ground-truth decision.

Tom: So the quality bar is concrete. The trajectory has to match the ground truth on significance and stay within one order of magnitude on the p-value itself.

Jane: Exactly. That filtering is what turns noisy teacher behavior into a clean warm start. They keep about 83 point 5 percent of the 4,611 teacher trajectories.

Lalam: And all of this happens without touching P-Bench itself. The evaluation set stays out of the training corpus by construction, which is what makes the final generalization claims meaningful.

Tom: So the SFT gives the model discipline — the shape of good statistical behavior, checking assumptions before committing to a method. But SFT alone doesn't push p-value accuracy far enough. The real gains come from the reinforcement learning stage, where the reward is tied to the actual statistical outcome.

Page 5 of the paper: Jane: So the reinforcement learning stage is where the accuracy gains come from, and the reward function is the star. Two components — a p-value closeness score and a conclusion correctness score — gated by a hard format constraint. If the trajectory doesn't contain reasoning, executable code, and a parseable final answer, the reward is zero.

Lu: The p-value component uses a z-score transformation, and that's a thoughtful design choice. Raw p-values are compressed near zero, so the difference between 0 point 5 and 0 point 6 looks identical to the difference between 0 point 1 and ten to the minus ten. But the first pair is barely a change in evidence, while the second spans orders of magnitude. The z-scale spreads out the region where differences actually matter.

Meng: They also avoid rewarding method choice directly, because multiple procedures can be defensible for the same question. Instead, the outcome-grounded reward carries that signal indirectly — pick the wrong test, get a p-value that deviates from the reference, and the score drops. The weights, 0 point 9 on the p-value and 0 point 1 on the conclusion, reflect that the conclusion check mainly verifies consistency with the reported p-value.

Lalam: That's the right call for scientific practice — you don't want the model penalized for a defensible alternative test. But it means the p-value comparison has to carry the whole inferential load, and the z-space scoring is what makes that work.

Jane: The algorithm is DAPO, with decoupled clipping and dynamic sampling. Groups where every rollout gets the same reward are discarded and re-sampled, so updates only happen where there's real signal. And the results table shows it working — Fisher-R1-14B beats GPT-5 point 4 on three of the four strict metrics, including 33 point 0 versus 30 point 5 on P-Hard pass@1.

Tom: The stability gain is striking too. The 7B backbone's standard deviation on P-Easy raw accuracy collapses from plus or minus 8 point 5 to plus or minus 1 point 6. So the model is both more accurate and more consistent across rollouts.

Lu: And the most important number is the gap between raw and strict for the frontier baselines. GPT-5 point 4 scores 58 point 3 raw on P-Hard pass@1, but only 30 point 5 strict. It gets the reject or fail-to-reject direction right almost twice as often as it produces a p-value close to the canonical analysis. Conclusion-only evaluation was overstating reliability.

Tom: So the reward design targets exactly that gap. But anytime a model improves this much, the obvious worry is memorization — did it just learn synthetic prompt templates? The paper addresses that head-on.

Page 6 of the paper: Jane: The answer to the memorization worry comes in two parts. First, the ablation. SFT plus DAPO is the best configuration on every metric. DAPO on the raw backbone does lift P-Hard strict pass@1 from 13 point 2 to 25 point 2, but it plateaus well below the full system's 30 point 6. And SFT alone improves pass@3 without reliably improving pass@1.

Lu: Right, so the SFT warm start gives the policy a broad, plausible distribution of solutions, and the RL sharpens it toward accurate p-values. The combination is what gets both. That's a nice illustration of why these two stages complement each other.

Meng: Then the generalization check. They embed prompts in semantic space and compare similarity between P-Bench prompts and the training pool. Train-to-train similarities form a tight high-similarity band, while eval-to-train similarities sit clearly below it. So the 425 benchmark tasks have no near-duplicates in the training corpus.

Jane: And that fits the construction of the two sets. Training is synthetic simulations; evaluation is real-world data from published analyses. The performance gain reflects transfer, not retrieval of memorized prompts.

Tom: The discussion is honest about limits too. P-Bench evaluates a single hypothesis test per task. Extending it to multi-test pipelines with multiple-comparison correction is the natural next step. And they flag the harder question of making agents justify assumptions explicitly and know when no single test is adequate.

Lalam: I want to highlight their framing around misuse. A more reliable statistical agent can catch method-driven errors before they reach the literature — that's the reproducibility win. But it can also lend false legitimacy to weak claims. So they release the benchmark and the model as evaluation and oversight tools, not as substitutes for human statistical review.

Meng: Which is reassuring given how much training data was synthetic. The whole claim is that synthetic verified tasks teach real-world judgment, and the similarity analysis is exactly the check you'd want to see.

Tom: All the pieces line up — the benchmark, the training signal, the evidence against memorization, and a clear sense of what's still missing. That's the picture worth walking away with.

Conclusion: Tom: So let's close the loop. The paper's central point is that current LLM agents look fluent at hypothesis testing but are statistically unreliable, and the field didn't notice because the benchmarks weren't measuring inferential validity. P-Bench fixes the measurement, and Fisher-R1 shows that training with verified rewards closes a large part of the capability gap.

Jane: The results are hard to argue with. A 14-billion-parameter open-weight model beating GPT-5 point 4 on the strict hard metrics — that's about the training signal, not the size of the model. The z-space reward and the SFT warm start are the ingredients that make the difference. And the stability gain, the standard deviation dropping from 8 point 5 to 1 point 6, is what makes it usable in practice.

Lu: For me, the lasting contribution is the verifiability pipeline. Every answer key is grounded in a logged execution of canonical code and audited by domain experts. That's what makes it possible to train on synthetic tasks and still trust the real-world evaluation — and it's a template other benchmark builders should follow.

Meng: And the benchmark design leaves an agent nowhere to hide. Seventeen method categories, realistic traps, task families requiring Cox regression, instrumental variables, and mixed-effects models — you can't default to a single recipe and score well.

Lalam: Bigger picture: as autonomous research agents get more ambitious, statistical reasoning becomes the constraint on whether we can trust them in high-stakes settings like clinical trials and policy evaluation. This paper points at that bottleneck and shows a path through it, while insisting that human statistical review stays in the loop.

Jane: And they release P-Bench and Fisher-R1 as open tools, so the next round can build on them — multi-test pipelines, uncertainty over method choice, knowing when no single test is adequate.

Lu: The opening example stays with me, though. A frontier model seeing outliers, noting the warning signs, and still reporting the linear result as significant. That's the failure mode we should all be watching for in the next wave of tools.

Tom: That's a good place to leave it. Thanks for listening, everyone — we'll see you for the next paper.