Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents

This paper, authored by Noam Koren, Roy Bar-Haim, and Abigail Goldsteen (IBM Research), addresses the under-explored problem of evaluating the quality of benchmarks used for task-oriented conversational agents. The authors state that Task-oriented conversational agents are evaluated using curated or automatically generated benchmarks, yet benchmark quality is rarely assessed. They note that Poor benchmarks may contain inconsistent tasks, simplistic scenarios, or limited policy coverage, leading to unreliable evaluations.

The paper introduces a reference-free framework that uses LLM judges to assess benchmark consistency, complexity, and policy coverage, while providing actionable diagnostics of weaknesses. The framework is validated by demonstrating agreement with independent human annotations and by evaluating benchmarks generated by LLMs of varying capabilities, as well as benchmarks subjected to controlled quality-degrading perturbations. Across domains and judge models, the proposed metrics consistently distinguish between benchmark quality levels, and the framework is shown to be applicable to manually curated benchmarks as well.

The authors motivate their work by observing that task-oriented conversational agents are increasingly deployed in customer support, travel, and enterprise workflows, and unlike static question-answering systems, they must interact over multiple turns, follow domain-specific policies, operate in real-world environments, and use external tools. While the common approach is to evaluate agents using curated benchmarks with realistic tasks, manual construction is costly, requiring domain expertise, scenario design, tool and database simulation, and specification of expected behavior. This has motivated growing interest in automatically generated benchmarks which forgo manual curation and its implicit quality control.

The central question raised is how should benchmark quality itself be evaluated? The authors point out that synthetic benchmarks may contain inconsistent tasks, simplistic scenarios, or limited coverage of the capabilities and constraints they aim to test, and prior work shows synthetic evaluation data can differ from human-curated data in representativeness, difficulty, and induced model rankings. They also note that benchmark quality issues are not limited to synthetic data: a review of 445 benchmarks found pervasive construct-validity problems even in widely used, manually curated benchmarks, and τ-bench itself has been shown to contain flawed and ambiguous tasks. The authors emphasize that policy-grounded evaluation of conversational-agent benchmarks remains underexplored and that undetected benchmark flaws may therefore lead to unreliable conclusions about agent capabilities.

The paper's four main contributions are explicitly listed:

1. We identify and study the problem of evaluating conversational-agent benchmarks.

2. We propose multidimensional metrics that assess consistency, complexity, and coverage with respect to the domain policy.

3. We validate the metrics using benchmarks generated by different LLMs, controlled quality-degrading perturbations, and human evaluations.

4. We demonstrate the metrics on a manually curated benchmark (τ3-bench).

The work assumes the structure of recent conversational-agent benchmarks including τ-BENCH, τ2-BENCH, and INTELLAGENT, which evaluate agents through dynamic interactions with an LLM-based user simulator. In this framework, a benchmark typically focuses on a specific domain, such as airline or retail, which defines the environment in which the agent operates, comprising a set of available tools and an associated database. Each domain also includes a policy that specifies the rules, constraints, and procedures governing agent behavior.

Each task is formally defined as t = (d t, e t, i t), where d t is the task description, including the user request; e t specifies the expected agent behavior and should be consistent with the policy P; and i t is the initial database state.

The paper introduces four LLM-judge-based metrics that evaluate benchmark consistency, complexity, and coverage, providing a comprehensive view of benchmark quality and difficulty.

The consistency metrics measure whether the task description, domain policy, and expected behavior are aligned and free of contradictions. Two alignment metrics are defined, each producing scores from 1 to 10 with higher scores indicating stronger alignment:

• Description–Expected Behavior Alignment: M desc(t) = J desc(d t, e t), which measures whether the expected behavior is consistent with the task description and adequately addresses the user's intended goals.

• Policy–Expected Behavior Alignment: M pol(t) = J pol(d t, e t, P), which measures whether the expected behavior complies with the domain policy for the described scenario and penalizes unsupported actions and omissions of policy-required steps.

Benchmark-level scores are computed by averaging task-level scores: S m(B) = (1/n) Σ M m(t).

The authors argue that a key aspect of conversational-agent evaluation is testing whether the agent correctly identifies and handles requests that conflict with domain policies and that a high-quality benchmark should include broad and systematic coverage of such policy-violating scenarios. They focus on violations "because they are more discriminative: a probe that only checks whether a policy item is relevant to a scenario would yield near-complete coverage for almost any benchmark, since nearly every task engages most policy items, causing such a signal to saturate."

Using an LLM judge J violate that identifies policy items violated by a task, the paper defines:

• Policy Violations per Task (complexity): S v count(B) = (1/B) Σ V(t), measuring task complexity as the average number of policy violations identified per task.

• Policy Violations Coverage: S v cov(B) = (1/N) k: T(p k) ≥ K, measuring the fraction of policy items violated by at least K tasks (K = 3 in experiments).

Policy items were extracted semi-automatically using TOOLGUARD, subsequently reviewed and revised by one of the authors.

To validate the metrics, the authors construct benchmarks of varying quality within the same domain using two complementary approaches: "generating benchmarks with LLMs of varying capabilities yields more realistic benchmarks and finer-grained quality differences, while the perturbation-based approach provides greater control over the differences in quality."

Synthetic benchmarks are generated using the INTELLAGENT framework for each target domain given a domain policy and database schema. The pipeline decomposes the policy into flows, builds a policy graph whose nodes represent policy items and whose edges model their likelihood of co-occurring in the same task, and for each task a subset of policies is sampled through a random walk over the graph and used to generate the task description and expected behavior. The scenario is then augmented with symbolic representations and instantiated entities.

To create benchmarks of varying quality, the authors replace GPT-4o in selected stages with LLMs spanning a broad range of capabilities, using generator capability as a proxy for expected benchmark quality. The benchmarks are generated using GPT-5.4, Claude-4.5-Sonnet, Llama-3.3-70B, Llama-3.1-8B, and Llama-3.2-1B, each containing 100 Airline tasks and 95 Retail tasks, yielding 975 synthetic tasks in total. The benchmarks are evaluated using three LLM judges: GPT-5.4, Claude-4.5-Sonnet, and Gemini-2-Flash.

Two controlled perturbations are applied:

• Expected Behavior Swapping: to break the alignment between the task description and the expected behavior, we swap the expected behavior in 20%, 40%, 60%, and 80% of the tasks.

• Cross-Domain Policy Swapping: to evaluate policy-related metrics, we replace the policy of one domain with that of another domain (e.g., airline tasks evaluated against the retail policy).

The authors define a meta-metric called the Benchmark Ordering Score, computed as the fraction of constraints satisfied from a partial order over benchmarks partitioned into three tiers: Top (GPT-5.4, Claude-4.5-Sonnet), Medium (Llama-3.3-70B, Llama-3.1-8B), and Small (Llama-3.2-1B). Results show substantial score gaps between benchmarks, with metrics consistently assigning higher scores to benchmarks generated by stronger models... across metrics, domains, and judges. Specifically, all metrics achieve perfect ordering (1.0) in the Airline domain, and a mean of 0.92 in Retail. Policy Violations Count achieves perfect ordering across all judges and domains. The overall mean ordering score is slightly higher for GPT-5.4 and Claude-Sonnet-4.5 than for Gemini-2-Flash (0.97 vs. 0.94). The authors conclude: These results provide strong evidence that our benchmark-quality metrics reliably capture quality differences among synthetic benchmarks.

Using Gemini-2-Flash as the judge, the Description–Expected Behavior Alignment metric "decreases consistently as the proportion of tasks with swapped expected behaviors increases. This pattern holds across both domains and all benchmark generators, demonstrating that the metric is sensitive to controlled degradation in task coherence. For cross-domain policy swapping, all three metrics decrease substantially when tasks are evaluated against a policy from the other domain. The consistent decline across domains and benchmark generators shows that these metrics detect mismatches between benchmark tasks and their associated domain policy."

A human annotator rated 50 tasks, sampled across benchmarks generated by all five models, along the three task-level measures in our framework. Using Kendall's τb correlation, all nine correlations (3 metrics × 3 LLM judges) are positive and statistically significant with values "ranging from 0.32 to 0.67, with raw p-values ranging from < 10−6 to 0.011. Agreement is strongest for Policy Violations Count, with correlations ranging from 0.55 to 0.67. The authors state: Overall, these results provide independent evidence that the LLM judges produce task-level rankings that align with human assessments."

As a case study using Policy–Expected Behavior Alignment, the analysis identifies five recurring issues in the generated expected behaviors: (1) Hallucinated workflow stages or unsupported actions; (2) Missing required information collection; (3) Missing explicit confirmation before an action; (4) Incorrect compensation handling; and (5) Forbidden actions. The authors note: These categories demonstrate how the metric can distinguish among different benchmark weaknesses rather than providing only an aggregate score. Illustrative examples show that low-scoring tasks expose concrete benchmark defects such as unsupported workflow invention, omission of required steps, and direct contradictions of policy rules.

The framework is also applied to the manually curated τ3-BENCH, evaluating all 50 Airline tasks with Claude-4.5-Sonnet and GPT-5.4 as judges, using metric prompts minimally adapted for structural differences from INTELLAGENT. Results show that the τ3-Bench scores broadly fall within the range observed for the lower-scoring INTELLAGENT benchmarks generated by Llama-3.1-8B and Llama-3.2-1B. The authors explain this partly by "differences in expected-behavior granularity: in τ3-Bench, the expected behavior typically specifies only the final outcome... whereas INTELLAGENT benchmarks generated by stronger models describe the full policy-compliant workflow. Additionally, τ3-Bench also shows substantially fewer policy violations per task (0.38–0.60) than the strongest INTELLAGENT benchmarks (1.5–2.3), indicating that top-generator INTELLAGENT tasks are considerably more adversarial and policy-complex. Finally, policy-violation coverage (0.31–0.46) leaves room for improvement, indicating that a substantial share of policy items are not challenged by any task in the benchmark."

The paper situates itself within the broader landscape of LLM-agent benchmarks covering tool use, planning, web interaction, software engineering, and general-purpose agents. The authors note that while prior work has attempted to validate synthetic benchmarks by comparing them to manually constructed references through model-ranking agreement, fidelity measures, and statistical similarity, these approaches provide only indirect evidence of benchmark quality and require suitable reference datasets. They also discuss Qian et al.'s approach for closed-form benchmarks, which requires evaluating many LLMs on each benchmark, which is considerably more costly for open-ended, task-oriented conversational settings. The authors claim: To the best of our knowledge, our work is the first to systematically evaluate benchmarks for conversational agents, with an approach that does not require a reference benchmark or running a large set of agents or LLMs on the benchmark being evaluated.

The paper concludes: "We introduced a framework for evaluating the quality of conversational-agent benchmarks, defining metrics that capture task-level consistency and complexity, and benchmark-level policy coverage. Beyond aggregate scores, these metrics provide actionable diagnostics, surfacing low-quality tasks and gaps in policy coverage. The validation demonstrates that the metrics reliably capture quality differences, remain sensitive to controlled degradation, and align with human judgments."

The authors acknowledge several limitations: the work focuses on conversational-agent benchmarks following τ-BENCH- and INTELLAGENT-style evaluation frameworks; the synthetic-benchmark experiments rely on the policy-driven INTELLAGENT generation pipeline, and other strategies, such as tool-driven generation, may produce different task distributions; and the paper does not consider non-conversational agent settings, such as code generation or computer interaction, which involve different interaction structures and evaluation challenges.

Based on the paper, here are specific improvements an AI system can make and what the improved system can do:

• Integrate a reference-free benchmark evaluation layer using LLM judges, instead of relying on indirect validation via agent performance or human-curated references.

• Add three multidimensional quality metrics for conversational-agent benchmarks:

• Description–Expected Behavior Alignment

• Policy–Expected Behavior Alignment

• Policy Violations Count and Coverage

• Use the evaluation layer to produce actionable diagnostics, not just aggregate scores, by identifying specific task-level flaws.

• Make the evaluation model-agnostic and judge-robust by aggregating judgments across multiple LLMs (e.g., GPT-5.4, Claude-4.5-Sonnet, Gemini-2-Flash) and testing consistency.

• Enable controlled perturbation detection to flag incoherent or mismatched benchmark tasks.

• Automatically score the quality of any conversational-agent benchmark without needing a reference benchmark or running many agents on it.

• Detect task inconsistency by measuring whether the expected agent behavior actually satisfies the user request described in the task.

• Detect policy violations by checking whether the expected behavior complies with the domain policy, flagging unsupported actions and missing mandatory steps.

• Measure benchmark complexity by counting how many policy items are violated per task on average.

• Measure policy coverage by identifying which policy items are never challenged or are insufficiently covered across the benchmark.

• Rank synthetic benchmarks by quality and distinguish benchmarks generated by stronger vs. weaker LLMs with high agreement with human judgments.

• Flag degraded benchmarks when expected behaviors are swapped or when tasks are paired with the wrong domain policy.

• Expose concrete benchmark weaknesses such as:

• Hallucinated workflow stages

• Missing required information collection

• Missing explicit confirmations before actions

• Incorrect compensation handling

• Forbidden actions

• Evaluate manually curated benchmarks like τ3-Bench and reveal under-tested policy areas and over-simplified expected behaviors.

• Provide per-task and per-policy diagnostic reports, enabling benchmark creators to fix weak tasks and fill policy-coverage gaps rather than merely receiving a quality score.