Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows

summary

Video file (mp4)

The gist

The gist The benchmark introduces Harness-Bench, a diagnostic protocol for evaluating configuration-level harness effects in realistic agent workflows.

In short

Harness-Bench is a diagnostic benchmark designed to evaluate how different system layers, or 'harnesses,' affect AI agent performance in real workflows. It tests 106 practical tasks across various models and harness configurations. The findings show that agent success depends on the model-harness pairing, not just the model alone, revealing where reasoning breaks down when tools and workspace states are misaligned.

Key concepts

Harness
The harness is the system layer that manages a complex AI agent's execution. It controls how context, tools, state (like workspace data), constraints, permissions, tracing, and recovery mechanisms work together to mediate between the base language model's raw output and its actual external actions.
Harness-Bench
This is a diagnostic benchmark used to study the effects of different harness configurations. It fixes the task environment while systematically varying which harness surrounds the model. It records detailed execution traces and artifacts to measure performance across these various setups.
Execution Alignment Failure
This occurs when an agent's logical reasoning becomes disconnected from its operational reality. Specifically, this means the model's internal thoughts no longer match what is happening in the external environment, such as tool feedback, changes in the workspace state, or verifiable output requirements.
Model-Harness Configuration Level
This refers to evaluating agent capability based on the specific combination of a base model and its surrounding harness. The research suggests that performance metrics should be reported at this combined level rather than attributing success solely to the underlying language model.

Terminology used across episodes

This episode discusses

The paper

Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows · Read on arXiv

Yilun Yao, Xinyu Tan, Chao-Hsuan Liu, Yaoming Li, Zhengyang Wang, Wenhan Yu, Zhewen Tan, Yuxuan Tian, Guangxiang Zhao, Lin Sun

Peking University

LLM agents are increasingly deployed as executable systems that use tools, modify workspaces, and produce concrete artifacts. In such workflows, performance depends not only on the base model, but also on the harness: the system layer that manages context, tools, state, constraints, permissions, tracing, and recovery. However, existing benchmarks typically abstract away execution, compare complete agent systems, or hold the harness fixed, making execution-layer variation difficult to study. We introduce Harness-Bench, a diagnostic benchmark for evaluating configuration-level harness effects in realistic agent workflows. Harness-Bench evaluates representative harness configurations across multiple model backends under shared task environments, budgets, and evaluation protocols, while preserving each harness's native execution behavior. The benchmark contains 106 sandboxed offline tasks constructed from practical agent-use patterns and manually reviewed for realism, solvability, oracle-checkability, and integrity. Each run records final artifacts, execution traces, usage statistics, and validator outputs, enabling analysis beyond final completion. Across 5,194 execution trajectories, we observe substantial variation in completion, process quality, efficiency, and failure behavior across model-harness pairings. These results suggest that agent capability should be reported at the model-harness configuration level rather than attributed to the base model alone. Our analysis further identifies recurring execution-alignment failures, where plausible reasoning becomes decoupled from tool feedback, workspace state, evidence, or verifiable output contracts. Harness-Bench provides a reproducible foundation for diagnosing and improving reliable, efficient, and auditable agent execution stacks.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows".

Jane: The gist The benchmark introduces Harness-Bench, a diagnostic protocol for evaluating configuration-level harness effects in realistic agent workflows.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: The paper we just discussed is "Harness-Bench: Evaluating Agent Harnesses across Models and Realistic Workflows." It really focuses on the harness as a variable, not just the model.

Jane: That’s right. The authors are showing that you need to look at the entire configuration of the agent system—the model plus its tools and state management—to understand performance accurately > Page one --- <ref:2605.27922#pg1>

Lu: They use this benchmark to show that when you have different models, they can perform very differently depending on how much control or structure is imposed by the harness > Page two --- <ref:2605.27922#pg1>

Meng: It means if we want reliable, efficient agent systems, we have to stop focusing only on the model weights and start looking at the entire execution stack setup > Page two --- <ref:2605.27922#pg1>

Lalam: The overall implication is that reporting agent capability should be tied to this model-harness configuration rather than just the base model alone > Page two --- <ref:2605.27922#pg1>

Tom: So, if you’re building an agent system, you need to evaluate it against a harness and see how that pairing performs across different models > Page one --- <ref:2605.27922#pg1>

Conclusion: Tom: So we’ve been looking at how different models can perform differently depending on the setup, but this paper is about that setup itself, focusing on something called a harness > Jane, what is the main thing they are trying to measure with this Harness-Bench thing?

Jane: Well, they're testing how much a specific tool or state management system—the harness—actually changes what an AI can do across different models > Lu, you know about how crucial that layer is for making sure the model’s brain actually connects to the real world actions > Meng, from your production side, when you deploy an agent system, does that harness configuration ever cause a weird failure mode that just doesn't show up in standard testing?

Meng: Exactly. Standard testing doesn't catch execution drift where the model thinks it’s doing something right because of its reasoning, but the harness messes up the workspace state or ignores a tool output > Tom, what kind of "execution alignment failure" are they seeing most often in these runs?

Tom: They find that plausible reasoning gets completely disconnected from tool feedback or what’s recorded in the workspace state > Lalam, you see if that decoupling happens when you try to run these one hundred six tasks > Lalam, what does that decoupling look like concretely?

Lalam: It looks like the model reasons about a plan, but when it tries to execute the tool call, it uses an outdated piece of information from the workspace or ignores a constraint set by the harness > Jane, so if we put this into practice for someone just listening to the show, what does that mean for how we trust these AI agents?

Jane: It means you can’t just look at one model and say it’s good; you have to look at the whole package—the model plus the harness—because that configuration determines if an agent is reliable enough to be used in a real workflow > Lu, what is the authors' big conclusion about how we should report on AI capabilities now?

Lu: They are saying capability isn't something you can just assign to the base model weights anymore; it has to be reported at the model-harness configuration level because that’s where the actual execution happens > Tom, so what does this imply for building better tools or interfaces for these agents?

More episodes

← Home