Deployment Decision Reliability: A Generalizability-Theory Framework for Sizing Long-Horizon Agent Evaluations

Deployment Decision Reliability: A Generalizability-Theory Framework for Sizing Long-Horizon Agent Evaluations

Vasundra Srinivasan, Independent Researcher, Stanford School of Engineering. arXiv:2608.11323v1 [cs.AI], 11 Aug 2026.

The paper's central finding is that Enterprise practitioners read agent leaderboards as if they ranked agent capability, but this reading is wrong. Across three open agent-trace benchmarks (TheAgentCompany, τ2-bench, and AppWorld), the authors show that the agent main effect accounts for less than 3% of total variance in every dataset and check type, while the agent-by-task interaction accounts for 7–23%. The paper concludes: Leaderboards rank specialization, not capability.

The authors apply a four-facet Generalizability Theory (G-theory) variance decomposition, modeling each observation as a binary success indicator Y atse at step s for agent a on task t with error-category label e. The fully crossed design decomposes expected score as: Y atse = μ + νa + νt + νs + νe + νat + νas + νae + νts + νte + νse + ε atse. Three-way and four-way interactions were omitted due to uneven cell counts and zero pilot estimates.

Three estimators were used and agreed to three decimal places: Henderson Method-I (closed-form ANOVA-based), REML via lme4, and a Bayesian binomial GLMM via bambi/numpyro run on an NVIDIA L4 GPU. The Bayesian fits used 4 chains × 1000 draws, completing in approximately 70 minutes on GPU.

Two reliability coefficients were computed: the generalizability coefficient Eρ2 = σa2/(σa2 + σδ2) for relative ranking decisions, and the dependability coefficient Φ for absolute-threshold decisions such as SLA gates.

1. Capability ceiling. The agent main effect σa2 is below 3% of total variance in every dataset and check type, and is exactly zero on τ2 db check and nl assertions. The agent-by-task interaction σa:t2 accounts for 7.5–12.5% of total variance, one to two orders of magnitude larger than σa2. The capability-gap ratio σa2/(σa2 + σa:t2) is below 0.001 on TheAgentCompany and below 0.19 on every τ2 cell. The pre-analysis hypothesis that σa2 would dominate was rejected.

2. Hard-task collapse. Aggregate Eρ2 does not predict hard-quartile Eρ2. On TheAgentCompany, aggregate Eρ2 = 0.986 falls to 0.869 on the hardest task quartile (Q4, mean success rate 0.066). On τ2 action checks, aggregate Eρ2 = 0.752 collapses to 0.000 in Q4, because σa2 = 0 among the hardest tasks in this check type. The paper states: When the production task mix is harder than the benchmark mix, the headline reliability number provides no useful guarantee.

3. Held-out reversal. Across 50 random 70/30 training/test splits, training-cell Eρ2 correlates negatively with held-out Eρ2 across 50 random splits (r = −0.90 on τ2). On τ2 action checks, the Pearson correlation between projected and held-out Eρ2 is r = −0.90 (Spearman ρ̂ = −0.91), with held-out reliability falling short of projection by 0.30 on average. The pattern is consistent across check types: r = −0.69 on db check, r = −0.89 on env assertions, and r = −0.44 on TheAgentCompany. The paper interprets: The training-cell projection systematically overstates the reliability a replication will recover.

4. Aggregate-versus-family asymmetry. The capability-gap ratio is stable at 0.35–0.40 across TheAgentCompany and AppWorld (0.38 on TheAgentCompany, 0.40 on AppWorld test normal, 0.35 on AppWorld test challenge), but per-family rankings invert (Spearman ρ̂ = −0.50, n = 3, across GPT-4o, GPT-4-Turbo, and Llama 3). The paper concludes: Population diagnostics transfer; family-level claims do not.

5. Variance ≠ frequency. Per-category σa:t2 and per-category failure-activation rates rank orthogonally (ρ̂ = −0.20 on MAST, n = 4, descriptive). The paper states: The categories where agents differ are not the categories where failures appear.

Reliability per Dollar is defined as RPD(a, B) = Eρ2(a, B)/C̄(a, B), where C̄ is mean inference cost per task observation. On τ2 action checks, accuracy ranking is Claude 3.7 Sonnet (83%), GPT-4.1 (81%), o4-mini (78%), but under RPD the order becomes Claude 3.7 Sonnet, o4-mini, GPT-4.1—o4-mini advances from third to second because its slightly lower accuracy is offset by its substantially lower per-turn inference cost. The accuracy and RPD rankings disagree on every τ2 check type except db check.

Using the MAST 14-mode failure taxonomy on the MAD dataset (1,242 LLM-judged traces from 7 multi-agent systems × 7 benchmarks), Method-I variance decomposition found σ2(trace × mode) accounts for 71.8% of total variance, σ2(trace) for 13.6%, and σ2(mode) for 11.3%. All system- and benchmark-level components are below 1.1%. The 70/30 trace-level hold-out (50 splits) gives average per-trace MAE = 0.261, confirming trace-level mode profiles are idiosyncratic. However, at the cell-aggregate level, mean activation rates predict with mean cell-mode MAE = 0.056 and Pearson r = 0.83—cell-level profiles generalise.

The paper presents a detailed example comparing GPT-4.1 and o4-mini on τ2 action checks. The leaderboard headline favors GPT-4.1 (81% vs. 78%), but DDR analysis shows: σa2 = 1.74% means the gap is dominated by task sampling; cost-aware layer inverts the rank favoring o4-mini under binding cost constraints; difficulty-conditional layer shows Eρ2 collapses to 0.000 in the hardest quartile; held-out layer shows the 0.752 figure is itself an overestimate. The DDR recommendation: "the leaderboard rank is real but small; o4-mini is preferred under any binding cost constraint; before either is deployed, run a domain-specific evaluation on production-representative Q4 tasks and report the held-out Eρ2 alongside the training-cell projection."

The paper introduces Deployment Decision Reliability (DDR), described as "a one-page reporting discipline that turns the variance-component table into five decisions an enterprise buyer can defend: how many tasks to sample, which difficulty bucket to target, which cost band to constrain on, which holdout protocol to demand from a benchmark vendor, and how to read a failure-mode taxonomy. DDR is explicitly not a new metric but a checklist that exposes the five findings above directly to the procurement decision."

Three generalizable takeaways are stated: (1) Stop reading leaderboard rank order as a procurement signal on its own—use published rankings as a starting hypothesis, then re-evaluate on production-representative task samples; (2) Demand a difficulty-conditional reliability number from your benchmark vendor—the aggregate Eρ2 is the wrong number when production tasks skew hard; (3) Always pair a training-cell reliability projection with a held-out cross-validated estimate—the held-out number will usually be worse.

The paper also notes the multi-model implication: If no agent uniformly wins, the procurement-correct decision is not which model do I pick, but which model do I route each task class to, framing the per-task-class σa:t2 table as a routing table.

Three limitations are acknowledged: (1) cross-dataset rank-correlation analyses are underpowered (three shared model families for TheAgentCompany-versus-AppWorld, four categories for τ2-versus-MAST); (2) the G-study treats each step within a trajectory as exchangeable, appropriate for AppWorld unit tests and τ2 reward checks but not for free-form trajectory segmentation; (3) the capability-ceiling finding is a statement about the current population of frontier agent harnesses and does not imply no future architecture will reopen the capability gap.

Datasets used: TheAgentCompany (175 enterprise tasks, 17 agents, 7,783 rows), τ2-bench (customer-service workflows, three frontier agents, 4 trials per task across 50 tasks per domain, five check types), AppWorld (750 enterprise-app tasks, 18 agent configurations, 79,650 unit-test rows), and MAD/MAST (1,242 traces with 14 binary failure-mode labels). All code, data loaders, fit artifacts, and reproducibility instructions are released under an open-source license at https://github.com/vasundras/agent-reliability-engineering-lab.

Improvement 1: Difficulty-Conditioned Reliability Reporting

The AI system will compute and report reliability metrics (e.g., Eρ2) separately for task-difficulty quartiles, not just aggregate. It will flag when aggregate reliability is misleading (e.g., Eρ2 = 0.986 aggregate vs. 0.869 on hardest quartile). The system can then warn users: Your production tasks are 2× harder than benchmark average; expected reliability drops from 0.99 to 0.87, and may collapse to 0.00 for the hardest 25%.

Improvement 2: Held-Out Validation Before Deployment

The AI system will automatically split evaluation data into training and held-out sets (e.g., 70/30), compute reliability on both, and report the held-out value as the conservative estimate. It will detect when training-cell projections overstate reliability (e.g., r = −0.90 between projected and held-out Eρ2) and refuse to recommend deployment based solely on training-cell numbers.

Improvement 3: Cost-Aware Model Selection

The AI system will rank models not by raw accuracy but by Reliability per Dollar (RPD = Eρ2 / mean inference cost). It will reorder candidates when cost differences are material (e.g., o4-mini moves from 3rd to 2nd because its 3% accuracy deficit is offset by 40% lower cost). The system will output: Under your 500/month inference budget, o4-mini achieves 0.74 reliability per dollar vs. GPT-4.1's 0.61—choose o4-mini.

Improvement 4: Task-Routing Recommendations

The AI system will use the agent-by-task interaction variance (σa:t2) to build a routing table, not a single-model recommendation. For each task class, it will identify which agent has the highest conditional success probability, even if that agent loses on aggregate. Example output: For billing disputes, route to Agent B (82% success) over Agent A (74%), despite A winning overall.

Improvement 5: Failure-Mode Generalization Checks

The AI system will separate trace-level failure profiles (idiosyncratic, MAE = 0.261) from cell-level profiles (generalizable, MAE = 0.056). It will refuse to make claims about individual traces but will confidently report aggregate failure-mode distributions across task types. It will flag when a failure category is high-variance but low-frequency (e.g., Hallucination appears rarely but drives 30% of agent differences).

Improvement 6: Benchmark Sufficiency Calculator

The AI system will compute the minimum number of tasks and trials needed to achieve a target reliability (e.g., Eρ2 ≥ 0.90) given observed variance components. It will output: You need 120 tasks × 5 trials to distinguish these agents at 0.90 reliability; your current 50-task benchmark only supports 0.75.

Improvement 7: Anti-Leaderboard Bias Warnings

The AI system will automatically detect when agent main effects (σa2) are negligible (<3% of variance) and warn: This leaderboard ranks task specialization, not general capability. Any rank difference smaller than 5% is within task-sampling noise. It will suppress rank-based procurement decisions when the capability-gap ratio is below 0.20.

Improved AI System Capabilities

The enhanced system can: (1) pre-deployment reliability audits with difficulty and cost conditioning, (2) multi-agent routing instead of single-model selection, (3) honest uncertainty quantification that distinguishes training-cell optimism from held-out reality, (4) failure-mode analysis that separates generalizable patterns from trace-specific noise, and (5) procurement-grade reports that answer How many tasks? Which difficulty? What cost? Which holdout? How to read failures?—directly addressing the five DDR decisions.

Enterprise practitioners read agent leaderboards as if they ranked agent capability. We show, across three open agent-trace benchmarks (TheAgentCompany, tau squared-bench, and AppWorld), that the agent main effect accounts for less than 3% of total variance in every dataset and check type, while the agent-by-task interaction accounts for 7-23%. Leaderboards rank specialization, not capability. We arrive at this through a four-facet Generalizability Theory variance decomposition, fit with three estimators (Henderson Method-I, REML via lme4, and a Bayesian binomial GLMM) that agree to three decimal places. Four further findings sharpen what the leaderboard is hiding. First, aggregate reliability collapses on the hardest task quartile: E rho squared on tau squared action checks falls from 0.752 to 0.000. Second, training-cell reliability negatively correlates with held-out reliability (r = -0.90 on tau squared), meaning the designs that look most reliable replicate worst. Third, population-level diagnostics transfer across enterprise benchmarks (capability-gap ratio stable at 0.35-0.40) but per-family agent rankings invert. Fourth, on the MAST failure taxonomy, trace-level mode profiles are idiosyncratic (MAE = 0.261) while cell-level profiles generalise (MAE = 0.056, r = 0.83). We package these into Deployment Decision Reliability (DDR), a one-page reporting discipline that turns the variance-component table into five decisions an enterprise buyer can defend. All code, data loaders, and fit artifacts are released under an open-source license.