Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks

arXiv:2604.24955 · cs.CL, cs.AI, cs.SE · Submitted 2026-04-27 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks".

Jane: As benchmarks grow in complexity, many apparent agent failures are not failures of the agent at all—they are failures of the benchmark itself: broken specifications, implicit assumptions,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: Welcome back everyone, and today we're talking about a paper that’s really tackling the problem of broken benchmarks in the AI agent world. We're looking at "Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks." This paper claims that many agent failures aren't actually problems with the agents themselves, but rather flaws in how those benchmarks are designed.

Jane: That makes so much sense, Tom; it sounds like they’re looking at the infrastructure behind the testing instead of just blaming the agents being tested. The paper introduces something called BENCHGUARD, which is this automated system that uses frontier LLMs to check all the different parts of a benchmark together.

Lu: What excites me about this is how they treat it as a consistency problem across four interlocking components: the instruction, the environment, the gold solution, and the evaluation logic. It's not just checking one thing in isolation; it’s seeing if all those pieces line up correctly <ref:2604.24955#pg2>.

Meng: From an engineering standpoint, I’m curious about how this framework handles the complexity of execution-based tasks. When you're dealing with agent traces and generated programs, how does BENCHGUARD manage that kind of input efficiently without slowing down the whole process?

Lalam: I think from a cultural perspective, Lu's point about treating it as a system consistency problem is really interesting because it suggests we need to shift our focus from just improving individual agents to rigorously validating the entire testing ecosystem. This framework could lead to better standards for what counts as a reliable agent evaluation <ref:2604.24955#pg1>.

Tom: Exactly, Lu, and that leads right into what they claim this automated auditing framework actually does. They propose using structured LLM protocols to cross-verify these artifacts and detect logical flaws that people often miss when they review things manually <ref:2604.24955#pg1>.

Jane: So, what's the core idea behind BENCHGUARD, in terms of what it’s designed to do for these benchmarks? Basically, it’s a way to systematically check for errors that stem from underspecified instructions or rigid evaluation scripts <ref:2604.24955#pg1>.

Paper summary: Lu: The authors hypothesize that many persistent errors come from solution fixation and the curse of knowledge, where creators assume implicit choices are obvious, leaving instructions incomplete <ref:2604.24955#pg1>. BENCHGUARD is designed to catch those gaps by using frontier LLMs as systematic auditors instead of just judges or subjects themselves <ref:2604.24955#pg1>.

Meng: I saw they mention the framework can audit fifty complex tasks with five frontier models for under fifteen dollars <ref:2604.24955#pg2,50 complex tasks with five frontier models for under>. That’s a pretty tangible cost, which is good for practical implementation, but does that cost translate to real-world reliability or just a quick check?

Lalam: The value proposition seems to be in the systematic nature of the audit; it’s about catching defects that prior human review simply missed across many different artifacts <ref:2604.24955#pg1>. This could significantly improve the quality control pipeline for building new agent benchmarks.

Tom: Speaking of quality control, let's look at what they actually found when they tested this framework on real benchmarks like ScienceAgentBench and BIXBench. The paper reports specific findings regarding the types of issues it caught.

Jane: They found that BENCHGUARD identified twelve author-confirmed defects in ScienceAgentBench and achieved an eighty-three point three percent alignment with expert-identified issues on the BIXBench Verified-fifty subset <ref:2604.24955#pg2>. That’s a pretty solid start for empirical validation.

Lu: It's important to remember that this framework operates by treating auditing as a cross-artifact consistency problem where misalignment among the instruction, environment, gold solution, and evaluation logic causes the defects <ref:2604.24955#pg2>. This holistic view is key to finding these systemic issues.

Meng: I’m thinking about that classification system they introduced—the taxonomy involving GT, EVAL, INST, and ENV categories—how does that structured labeling help when you're trying to pinpoint exactly where the benchmark specification is failing?

Lalam: The taxonomy allows for a very specific diagnosis; instead of just saying "this task failed," you can say "this failure falls under INST-INCOMPLETE" or "GT-LOGIC," which gives experts a much clearer signal on what kind of fix is needed <ref:2604.24955#pg1>.

Tom: And that structure leads us into the conclusion of this discussion, where we look at the bigger picture implications of this work. The paper introduces BENCHGUARD as a tool to ensure benchmarks measure what they claim they are measuring.

Paper summary: Jane: So, when we look at the title, "Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks," it really frames this whole effort as establishing a new layer of quality assurance for the agent testing landscape.

Lu: I see this having massive implications for how we develop AI evaluation suites; it suggests that relying solely on human review might not be enough to catch these subtle, cross-artifact errors <ref:2604.24955#pg1>.

Meng: For me, the practical impact is in creating a more reliable feedback loop for researchers; if we can automate the detection of flaws in the testing infrastructure, it streamlines the process of iteration and refinement for agent development.

Lalam: I think the most profound implication is establishing a path toward AI-assisted benchmark development itself, where frontier models become participants in validating the evaluation infrastructure they are being tested against <ref:2604.24955#pg2>.

Tom: So, to wrap up this part of our discussion on "Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks," it seems the authors are proposing a concrete, automated method using structured auditing to catch deep flaws in benchmark design itself.

Jane: That's right; they are moving beyond just evaluating agent performance and focusing on making sure the tests themselves are sound. It’s a practical step toward building more trustworthy AI evaluation systems.

Lu: This framework shows that treating benchmark creation as a complex system rather than just a set of isolated tasks opens up new avenues for identifying these kinds of systemic weaknesses <ref:2604.24955#pg2>.

Meng: It’s interesting how they show model-specific patterns, like GPT-five point four being the only model detecting evaluation bias errors, which tells us that robustness varies across different AI architectures <ref:2604.24955#pg1>.

Lalam: If we look at this through the lens of our work here at Lalam, it suggests that by applying these kinds of rigorous, cross-artifact checks to our own development cycles, we can ensure that the cultural standards and safety guardrails embedded in our models are actually being tested correctly <ref:2604.24955#pg1>.

Tom: Fantastic discussion so far. We’ve looked at what this paper is all about and why it matters for the reliability of agent testing. Next up, we're going to talk a bit more about what this means for the future of benchmark design itself.

Conclusion: Tom: So we've been looking at how this paper, "Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks," works by treating benchmark creation like a system consistency problem. Jane, can you simplify what that means for our listeners?

Jane: Absolutely, Tom; it basically means instead of just checking if an agent did a task right, they are using AI to check if all the parts making up the test—the instructions, the environment setup, and even the answers provided—actually make sense together. Lu, from your perspective at Tsinghua, what's your take on this approach to testing infrastructure?

Lu: I think it's a fascinating way to look at it because it treats benchmarks not as static problems but as dynamic systems that need constant internal verification. The idea of cross-referencing all those artifacts with structured LLM protocols opens up possibilities for building much more robust and self-correcting evaluation frameworks down the road.

Meng: I'm thinking about the practical side; how does this systematic check translate into something we can use in our actual engineering pipelines? We need to know if this adds significant overhead or if it’s a worthwhile investment for quality control.

Lalam: From my perspective as an LLM, the most impactful vision here is how this framework can fundamentally improve the culture around AI development by making the testing infrastructure itself more transparent and reliable. If we can automatically flag where instructions are incomplete or where evaluation logic is biased, it helps us build systems that are inherently fairer and more trustworthy for everyone.

Tom: That's a powerful idea, Lalam; moving toward a culture where the tools we use to test AI are as rigorously checked as the AI models themselves. Jane, what do you see as the bigger picture impact of this work on how we view agent evaluation?

Jane: I see it shifting our focus from just chasing better agent performance to ensuring that the benchmarks used to measure that performance are actually accurate and well-defined. It's about building a more reliable foundation for evaluating AI capabilities in general.

Lu: And the fact that they use a taxonomy of errors—like GT, EVAL, INST—shows they’re thinking about diagnosing specific types of flaws rather than just getting a general pass or fail. That level of detail is where the real innovation lies for future research into evaluation design.

Meng: I'm still focused on implementation details; if this auditing system can be automated and cost-effective, it could become a standard operating procedure for any organization building complex AI agents, which is what I'm looking at right now.

Lalam: Exactly; this work suggests that by automating the validation of our evaluation logic, we can create a feedback loop that is self-improving across the entire ecosystem. It's about making sure our methods improve themselves as we go.

Tom: Wow, from the hosts, it sounds like this paper isn't just about fixing specific agent bugs; it’s proposing a whole new way to build better AI evaluation tools and a more rigorous culture around testing. So, where do we go from here with this kind of automated auditing?

Xinming Tu, *Tianze Wang, *Yingzhou (Minta) Lu, Kexin Huang 2, *Yuanhao Qu 2, Sara Mostafavi 1,3,*

Allen School, University of Washington

cs.CL, cs.AI, cs.SE

Submitted: 2026-04-27

Updated: 2026-10-02

Comments: Camera-ready version for COLM 2026. 24 pages

Code: https://github.com/harbor-framework/harbor

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: As benchmarks grow in complexity, many apparent agent failures are not failures of the agent at all—they are failures of the benchmark itself: broken specifications, implicit assumptions, and rigid

Key concepts

BENCHGUARD
A novel automated auditing framework designed for task-oriented agent benchmarks. It uses frontier LLMs as auditors to cross-verify all benchmark components—like instructions and solutions—to detect hidden logical flaws.
Cross-Artifact Consistency Problem
The core problem BENCHGUARD solves is checking if all parts of a benchmark (instruction, environment, gold solution, evaluation logic) are consistent with each other. It treats auditing as finding inconsistencies across these four interlocking artifacts.
Error Taxonomy
A structured system for categorizing benchmark flaws into specific types like GT (Ground Truth), EVAL (Evaluation), INST (Instruction), and ENV (Environment). This taxonomy helps researchers understand where the most common bugs occur, such as instruction incompleteness.
Collaboration Boundary
The point where human experts and automated auditing tools work together. Sometimes, automated tools find a flaw that humans don't flag because the expert knows the flaw is intentional or methodologically motivated.

Terminology

Summary

As benchmarks grow in complexity, many apparent agent failures are not failures of the agent at all—they are failures of the benchmark itself: broken specifications, implicit assumptions, and rigid evaluation scripts that penalize valid alternative approaches. This paper introduces BENCHGUARD, a novel automated auditing framework designed to systematically cross-verify the coupled artifacts of task-oriented execution-based agent benchmarks using frontier LLMs as auditors.

The Gist

BENCHGUARD is the first automated auditing framework for task-oriented, execution-based agent benchmarks that cross-references all benchmark artifacts via structured LLM protocols to detect systemic logical flaws invisible to human review.

How it works

BENCHGUARD operates by treating benchmark auditing as a cross-artifact consistency problem involving four interlocking components: the natural-language instruction, the environment, the reference gold solution, and the evaluation logic. The core of its definition-level audit involves a single consolidated LLM call employing six phases of structured reasoning: (1) Task Understanding, (2) Ground Truth Correctness, (3) Evaluation Logic verification, (4) Task Specification check for underspecified requirements, (5) Environment flagging runtime issues, and finally, (6) Consolidation which includes deduplication via the one-fix test and atomicity verification via the split test.

The framework utilizes a structured LLM prompt template that mandates an expert auditor role. This protocol prioritizes instruction-level defects when ambiguity exists between instruction and ground truth problems, reflecting the hypothesis that the root cause of most benchmark bugs is an underspecified or ambiguous instruction. Findings are annotated with a severity level (Critical/High/Medium/Low) and a confidence score (Confirmed 0.8–1.0, Likely 0.55–0.79, Possible 0.3–0.54), suppressing findings below a confidence of 0.3 to ensure high-quality triage signals for human experts rather than noise that requires manual review for every finding.

Error Taxonomy and Validation

The framework introduces an empirically grounded taxonomy of four top-level categories: GT (Ground Truth), EVAL (Evaluation), INST (Instruction), and ENV (Environment). The full taxonomy includes 14 subcategories, such as GT-LOGIC, EVAL-JUDGE-BIAS, INST-INCOMPLETE, and ENV-PATH. This taxonomy is iteratively refined through audits of ScienceAgentBench and BIXBench. For instance, the paper notes that INST-INCOMPLETE consistently dominating across all models in BIXBench findings. The framework's effectiveness is validated by its performance on two prominent benchmarks: it identified 12 author-confirmed defects in ScienceAgentBench and exactly matched 83.3% of expert-identified issues on the BIXBench Verified-50 subset, catching defects that prior human review missed entirely.

Audit Modes and Diagnostic Evidence

BENCHGUARD supports a tiered input system offering increasing diagnostic power: Minimal (instruction + tests) for basic checks; Definition-level (+ gold program) for full cross-artifact auditing; and Execution-Level audit when agent traces are available. The execution-level audit appends the agent’s generated program and evaluation log to the context, allowing it to exercise code paths invisible to static review. This mode significantly improves recall, as shown by Opus 4.6 on ScienceAgentBench improving from 83.3% to 91.7%.

Empirical Results and Collaboration Boundary

The empirical validation across five frontier LLM backends demonstrated substantial but imperfect overlap in detection sets, with the ensemble achieving an 83.3% exact alignment on BIXBench issues. The analysis revealed model-specific patterns, such as GPT-5.4 being the only model detecting EVAL-JUDGE-BIAS errors, suggesting sensitivity varies across model families. Furthermore, the paper explores a Collaboration Boundary, illustrating where human and automated auditing complement each other; for example, in Case 6 of BIXBench, all five models flagged a defect (gold program silently dropping two samples), yet the expert review did not annotate it because the gold-program author knew the removal was methodologically motivated. This suggests that experts can over-correct where automated auditing shows restraint, highlighting a new dimension of problem detection specific to execution-based benchmarks.

Practical Application and Future Direction

The framework is designed to be practical, costing under a modest amount—a full audit of 50 complex bioinformatics tasks costs under [15 dollars]—making it a practical and valuable complement to human review. The ultimate goal is not just post-hoc remediation but integrating auditing into the benchmark construction process itself, enabling AI-assisted benchmark development, where frontier models serve not only as subjects of evaluation but as active participants in validating the evaluation infrastructure itself. This aims to establish reliable evaluation at scale by ensuring benchmarks measure what they claim to measure.

Improvements for AI systems

Here are specific improvements for AI systems based on the BENCHGUARD framework:

  1. Enhanced Benchmark Integrity and Quality Assurance:

  2. Systematic Detection of Cross-Artifact Inconsistencies (The Hidden Bug Detector):

  3. Automated Benchmark Revision and Repair Pipeline:

  4. Context-Aware Benchmarking (Incorporating Execution Traces):

  5. Model-Specific Weakness Profiling for Targeted Benchmark Hardening:

  1. Enhanced Benchmark Integrity and Quality Assurance:

The system can automatically audit the entire evaluation infrastructure of any agent benchmark before publication or widespread use. This moves quality assurance from a post-hoc, manual review process to a systematic, automated gate.

The improved AI system will identify and flag structural flaws in execution-based benchmarks—such as incorrect data handling logic (GT-DATA), faulty evaluation scripts (EVAL), or underspecified instructions (INST)—that are invisible to static label auditing.

  1. Systematic Detection of Cross-Artifact Inconsistencies (The Hidden Bug Detector):

The system will move beyond checking components in isolation. It will perform a definition-level audit that cross-references the four core artifacts (instruction, ground truth program, evaluation script, environment configuration) to detect logical inconsistencies that arise only from their interaction.

Specifically, it can catch:

  • Instruction vs. Reference Solution Mismatches: Detecting when an instruction points to a wrong file or format while the reference solution uses a different one (Case 1).

  • Evaluation Logic vs. Instruction Mismatches: Identifying cases where an evaluator checks for drug names when the instruction explicitly requested SMILES strings (Case 2).

  • Implicit Assumption Detection: Flagging underspecified requirements in instructions that lead to gold programs implementing incorrect interpretations of vague specifications (INST-INCOMPLETE).

  1. Automated Benchmark Revision and Repair Pipeline:

The system will generate actionable, categorized findings with severity and confidence scores, allowing developers to directly repair the benchmark artifacts rather than just reporting errors.

The improved AI system can:

  • Provide precise recommendations for fixing specific lines of code in the gold program or modifying evaluation scripts.

  • Use the error taxonomy (GT/EVAL/INST/ENV) to guide targeted remediation efforts, focusing on high-impact defects like fatal specification errors (Critical severity).

  1. Context-Aware Benchmarking (Incorporating Execution Traces):

The system can upgrade its auditing capability by incorporating agent execution traces or solutions as diagnostic evidence during the audit process. This allows for an execution-level audit that detects failures in runtime behavior, not just static logic.

The improved AI system can:

  • Identify edge cases and subtle scoring errors that only manifest when agents execute their code against the evaluation environment (e.g., detecting a tolerance issue only when a correct output is scored differently).

  • Improve recall by identifying defects that are missed by definition-level audits alone, as seen in the improved recall metrics on ScienceAgentBench.

  1. Model-Specific Weakness Profiling for Targeted Benchmark Hardening:

The system can profile which types of errors specific frontier LLMs are uniquely sensitive to (e.g., GPT-5.4's sensitivity to EVAL-JUDGE-BIAS).

The improved AI system can:

  • Identify which benchmark artifacts cause failure modes that are only exposed by certain model architectures, informing future benchmark design to ensure robustness across diverse agent capabilities.

  • Determine the Collaboration Boundary between human experts and automated audits by flagging findings where automated systems show restraint (precision) versus expert revisions (alignment), helping developers understand where human intuition is superior versus where automation provides critical structural checks.

Abstract

As benchmarks grow in complexity, many apparent agent failures are not failures of the agent at all---they are failures of the benchmark itself: broken specifications, implicit assumptions, and rigid evaluation scripts that penalize valid alternative approaches. We propose employing frontier LLMs as systematic auditors of evaluation infrastructure, and realize this vision through BenchGuard, the first framework explicitly designed for joint cross-artifact auditing of execution-based agent benchmarks. BenchGuard cross-verifies all benchmark artifacts via structured LLM protocols, optionally incorporating agent solutions or execution traces as additional diagnostic evidence. Deployed on two prominent scientific benchmarks, BenchGuard identified 12 author-confirmed issues in ScienceAgentBench---including fatal errors rendering tasks unsolvable---and exactly matched 83.3% of expert-identified issues on the BIXBench Verified-50 subset, catching defects that prior human review missed entirely. A full audit of 50 complex bioinformatics tasks costs under USD 15, making automated benchmark auditing a practical and valuable complement to human review. A preliminary native-format audit of ProgramBench further demonstrates cross-format applicability. These findings point toward AI-assisted benchmark development, where frontier models serve not only as subjects of evaluation but as active participants in validating the evaluation infrastructure itself.

Sources

Related papers