2608.07346-A2E: An End-to-End Agent Auditing Engine

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "A2E: An End-to-End Agent Auditing Engine".

Jane: The paper was written by Haoning Wang, Mingxun Zhang, Chenyue Yu, Yingjun Shang, Xia Hu et al. from Shanghai Artificial Intelligence Laboratory.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary: Tom: Alright, we've got a really interesting one today. This is a paper about evaluating the frameworks that actually run eye agents, not just the models themselves.

Jane: And that's a distinction that matters a lot more than people realize. The paper is built around a pretty simple observation: when you deploy an agent, the harness — the software that handles prompts, tools, and execution loops — can change results dramatically, even with the exact same model underneath.

Tom: So they built this system called A2E, an Agent Auditing Engine, to measure those differences systematically. It's three layers: a task layer that standardizes how benchmarks and harnesses talk to each other, a monitor layer that records every step the agent takes, and an evaluation layer that scores the whole trajectory, not just the final answer.

Jane: The key finding really jumped out at me. They ran nine different agent frameworks against twenty-three benchmarks using the same model, and there was no winner. No single harness dominated across all tasks. A framework that crushed one benchmark could be at the bottom on another.

Tom: Right, and that's the kind of result that makes people in the field uncomfortable, because it means you can't just pick a framework once and forget about it. Your choice has to depend on what tasks you're actually trying to solve.

Jane: And there's another striking finding — the final answer correctness barely varied across harnesses. The scores were all bunched between roughly 0 point 57 and 0 point 68. But when you look at planning quality, token efficiency, tool use, those varied wildly, like three and a half times the token cost between the most and least efficient.

Tom: So if you only measure whether the agent got the right answer, you'd think all harnesses are basically the same. But if you look at how they got there, you see enormous differences in cost, efficiency, and behavior.

Jane: Exactly. And that's why they designed the evaluation around the full lifecycle — reasoning, action, final answer, and runtime quality. They even have a case study with two trajectories on the same task where one used ten thousand tokens and the other used ninety-six thousand, and the cheaper one actually solved the problem correctly.

Tom: That's a pretty stark demonstration. So the paper's really saying: stop evaluating agents as if they were just models. The harness is part of the system, and it deserves its own measurement infrastructure.

Jane: And that's what A2E is trying to provide. An end-to-end engine that makes it practical to run these large-scale comparisons, with standardized traces, a database for results, and metrics that actually diagnose where agents go wrong.

Tom: So a quick question before we go deeper — how do they actually get all these different frameworks to talk to the same benchmarks? Because that's usually where these projects die.

Jane: Oh, that's the Agent Task Protocol. And that's exactly what we should talk about next, because it's the cleverest part of the design. Let's look at the actual paper and dig into the details.

Page 1: Tom: So we're on page one now, the abstract and the opening figure. And honestly, the figure alone tells you most of the story.

Jane: It does. There's this "petal" chart showing how much each metric varies across the nine harnesses. And correctness is this tiny little petal, spanning maybe 0 point 57 to 0 point 68. But planning alignment, tool invocation, token usage — those petals are huge.

Lu: That's actually the most striking visual in the whole paper. You see it immediately — if you only measure correctness, the nine frameworks look nearly identical. But the other petals open up like a fan, showing real differences in how agents plan and act.

Tom: And that's the whole thesis in one picture, right? The outcome layer barely moves, but the process layer is full of variation.

Jane: Right. And the abstract also explains the universal adapter idea. They have this Agent Task Protocol, or ATP, that lets all twenty-three benchmarks pair with all nine agent frameworks without writing any per-combination integration code. So instead of writing two hundred seven adapters, you write two protocols.

Lu: That's a big deal for practical research. In most labs, if you want to test three harnesses on four benchmarks, you're looking at twelve integrations, and each one is a little different, a little buggy, and a little out of date.

Tom: And that's exactly the pain they're describing in the introduction — existing tools either handle orchestration or observability, but not both, and the integration work never ends as interfaces evolve.

Jane: Yeah, they contrast with Inspect eye, which is good at sandboxed execution and scoring but needs harness-specific adapters, and Phoenix, which gives you observability but doesn't run benchmarks end to end. A2E is trying to be the lightweight substrate underneath both.

Lu: And there's a nice detail in the abstract — they call the monitoring "automatically instrumented." You don't add logging code to each agent. The monitor observes the execution through the framework's natural extension points and produces standardized traces.

Tom: So you get the trajectory without contaminating it. That's important because if you modify how an agent runs just to observe it, you might change the behavior you're trying to measure.

Jane: Right, they're very concerned about trajectory fidelity. And that's a theme that runs through the whole paper — the trace should be a faithful record of what the agent actually did, not a reconstruction from logs.

Lu: The other thing I noticed on this page is the petal chart includes metrics like prompt injection resistance and harmful actions. So they're not just measuring efficiency — they're measuring safety and robustness as first-class properties.

Tom: And those also show variation across harnesses, right?

Jane: They do. Some frameworks are better at resisting injection attacks or avoiding harmful actions, even with the same model. That suggests the harness itself matters for safety, not just the model.

Lu: Which is a pretty important finding for anyone deploying agents in production. If your harness choice affects safety properties, that's not a detail you can ignore.

Tom: So this page really sets up the stakes — evaluation infrastructure is the bottleneck, and the process matters as much as the outcome. But how does the actual engine work? Let's look at the system overview next.

Page 3: Jane: So we're moving into the actual architecture now. Page three has the introduction and the start of the overview section, and it really frames why this whole infrastructure problem exists.

Tom: The introduction makes a strong claim — as models improve, the harness increasingly determines overall system performance. That's a big statement, and the paper backs it up later with experiments.

Lu: It's also a shift in how people should think about benchmarks. When you evaluate a model in isolation, you're not seeing how it behaves in deployment. The harness changes the system prompt, the tool interfaces, how context is managed, the execution policy. All of that shapes the final outcome.

Jane: And the paper's answer to that problem is this three-layer design. Task layer on the bottom, monitor layer in the middle, evaluation layer on top. The task layer standardizes benchmarks and agent integration, the monitor captures traces, and the evaluation layer scores everything.

Tom: One thing I appreciated is how they talk about the benchmark tree. Benchmarks are organized along time, category, and difficulty. So you can distinguish saturated benchmarks from fresh ones, compare across domains, and separate easy from hard tasks.

Lu: That's useful because the field tends to over-fit to a few popular benchmarks. If you track when a benchmark was released and how hard it is, you get a more nuanced picture of whether agents are actually improving or just memorizing.

Tom: And then there's this idea of an execution-support bundle — the sandbox definition, the task dataset, the experiment configuration, and the runtime environment all packaged together. That's what makes experiments reproducible.

Jane: Right, because if you want to compare harnesses fairly, they all need to run in the same conditions. That's the only way you can attribute differences to the harness itself rather than to some environmental mismatch.

Lu: There's another nice touch in the monitor layer — they distinguish between representative agents, like Creweye or Smolagents, which are ready to run, and SDK-based agents, which are built with development kits like LangChain or LangGraph. But both get normalized into the same access abstraction.

Tom: So you can treat a simple agent and a complex multi-agent system through the same interface?

Jane: Exactly. And the monitor loop then captures the full reasoning-action-observation cycle as an ordered sequence. R1 to A1 to O1 to R2 and so on. That preserves both the final outcome and every intermediate decision that led to it.

Lu: I like that they stream everything to a centralized server. It's not just logs written to files that are hard to query. It's structured data in a database, which makes later analysis and visualization much easier.

Tom: And the evaluation layer combines two families of evaluators — rule-based ones for things like token counts and success rates, and LLM judges for qualitative dimensions like reasoning quality and safety.

Jane: Right. Some things you can measure exactly, and some things need a judgment call. The design keeps both, and both write their results back to the same database.

Lu: So this page is really about the philosophy — evaluation should be lifecycle-aligned, database-backed, and extensible. Those three principles drive everything else in the paper.

Tom: And it's those principles that let them run a thousand-plus scored runs and make sense of it. But I want to know more about the monitor itself — how do they actually capture what the agent is doing without breaking it?

Jane: That's exactly where we're headed. The monitor layer is what makes the whole system trustworthy, and it's got some clever mechanics around spans and instrumentation.

Page 5: Tom: So we're on page five now, which is where the monitor layer gets explained in real depth. And the key word here is "span."

Jane: Right. They adopt the OpenTelemetry span model. A span represents an operation with a start time, an end time, a status, and context. And spans nest inside each other, creating a tree that mirrors the agent's execution structure.

Lu: That's elegant because agent execution naturally forms a hierarchy. You have a top-level agent span, then inside it you have reasoning chains, model calls, tool invocations. Each tool call might have its own sub-spans. The tree structure preserves both the sequence of events and their causal relationships.

Tom: So it's not just a flat list of things that happened. It's a record of what triggered what.

Jane: Exactly. And the instrumentation is divided into three layers that separate concerns. The semantic layer defines what agent behaviors mean — agent, chain, model call, tool, skill. The span layer records when things happen and how they're related. And the SDK layer maps each framework's specific mechanisms onto that shared vocabulary.

Lu: That separation is what makes the whole thing extensible. When you add a new framework, you only write the SDK adapter for it. You reuse the semantic definitions and the span structure. So monitoring coverage grows without redefining how agent behavior is represented.

Tom: And there's a stronger claim buried in here too — that span-based tracing gives you information you can't recover from the final response alone. Duration tells you where time goes, status tells you where things fail, and the parent-child relationships tell you which reasoning path was followed.

Jane: That phrase "the trace shows which reasoning path was followed" is really the core promise. It's not just that you know the agent failed — you know exactly at which step and for what reason.

Lu: One practical implication is that you can compare two runs and see where they diverge. Two agents might both succeed, but one might call a tool ten times while the other calls it twice. The trace makes that visible in a structured, queryable way.

Tom: And that structure is what later enables the lifecycle-aligned evaluation, because each metric can be attached to the part of the execution it's meant to assess.

Jane: Right. The monitor provides the raw material — carefully organized spans that capture the full execution. And then the evaluation layer can decide how to interpret them.

Lu: There's also a nice note that framework-specific differences are resolved before the trace is organized. So the higher-level execution flow isn't obscured by the quirks of any particular SDK.

Tom: So monitoring is the foundation, but it only captures what agents do. It doesn't define how tasks are presented to agents. That's where the task layer comes in, and the Agent Task Protocol. Let's look at that next.

Page 7: Jane: So page seven has the task layer, and this is where the Agent Task Protocol gets defined properly. And the paper is careful to note that ATP is an internal software protocol, not a network protocol.

Tom: Right, it's a shared interface between the benchmark and the harness, not something that goes over the wire. And the design uses four objects: TaskInput, AgentBinding, AgentRunner, and TaskTrace.

Lu: The TaskInput stores the instruction, the state, the expected actions or outputs, metadata, and optionally a sandbox spec. And the AgentBinding provides the tool schemas, the actual tool execution, and prompt construction. So the binding is what adapts a benchmark's semantics to whatever the harness expects.

Tom: And then the AgentRunner is what executes each task. It runs the control loop and returns a TaskTrace, which has the run status, the final answer, and the ordered list of tool calls.

Jane: That separation is the critical design choice. The benchmark adapter creates the TaskInput and the AgentBinding. The harness supplies the AgentRunner. They never need to know about each other's internals.

Lu: And that's what makes the m-by-n grid possible. You have twenty-three benchmarks and nine harnesses, and you don't need two hundred seven adapters because the boundary is standardized on both sides.

Tom: The paper also lists the current harness registry, and it's quite the collection — Agno, AutoGen AgentChat, CrewAI, Google ADK, LangGraph, LlamaIndex, Openeye Agents SDK, Smolagents, and the Anthropic Python SDK. Nine frameworks across the major approaches people actually use.

Jane: And there's a nice honest detail — the paper says registry support doesn't imply that every framework-benchmark pair has passed end-to-end validation. Some combinations might not work yet, and that's okay.

Tom: On the benchmark side, they group twenty-three benchmarks into four task areas: coding, conversational, research, and computer-use. And they support three kinds of tasks: text-only, tool-use, and sandboxed tasks with a container.

Lu: I appreciated how they break down the trajectory generation path. The CLI samples forty tasks by default, and each run gets a unique identifier that records the framework, model, dataset, seed, and task IDs. That metadata is what makes reproduction possible.

Tom: And they record both a normalized TaskTrace and a full span tree. The trace identifier links them together. So you get the clean summary and the detailed execution record in one place.

Jane: That dual recording is important. The task trace is easy to evaluate — it has status, answer, tool calls, timing. But if you need the full detail, the span tree preserves everything at the framework level.

Lu: And the separation also means evaluation doesn't need direct access to the runtime. You can evaluate offline, re-evaluate later, add new metrics without rerunning a single experiment.

Jane: That's the key feature that makes the whole system scalable — evaluation is decoupled from execution. And speaking of evaluation, that's exactly what's on the next page we're about to discuss.

Page 9: Tom: So now we're at the evaluation layer, and this page introduces the lifecycle-aligned taxonomy. It's the intellectual heart of the paper, I think.

Jane: It really is. They organize metrics into four stages: Reasoning, Action, Final Answer, and Runtime Quality. And each stage has its own dimensions. Reasoning splits into Task, Flow, and Logical — understanding the objective, planning completeness, and coherence. Action splits into Tool, Skill, and Memory.

Lu: So you can pinpoint where an agent fails. Did it misunderstand the task? Did it plan poorly? Did it use the wrong tool? Did it forget something from earlier context? The taxonomy gives you a diagnostic address for every failure mode.

Tom: And then Final Answer has two dimensions — correctness and task completion. Those are subtly different things. An agent might give a correct-sounding answer but not actually complete the underlying task. Or it might complete the task but present a poor final response.

Jane: That's a distinction that standard benchmarks almost never make. And then Runtime Quality spans the whole trajectory — efficiency and safety. It measures things like token consumption, latency, cost, and operational risks like prompt injection.

Lu: The phrase they use is "lifecycle-aligned" because each metric is registered under the stage it's intended to assess. And that's what gives the evaluation its diagnostic power. You're not just asking "did it work?" — you're asking "where in the lifecycle did things go right or wrong?"

Tom: The extensibility story is clean too. The taxonomy separates where a property is evaluated from how it's measured. So you can add a new metric implementing LLM-judge scoring, or a deterministic rule, or an environment verifier, and register it under an existing dimension without touching the harness or the benchmark runner.

Jane: And because the taxonomy is high-level and the metric catalog is open, it can evolve as agents get new capabilities or as new safety requirements emerge. The structure stays stable, but the contents can grow.

Lu: There's also the scalability argument. All the trajectories live in a database with explicit relationships between benchmarks, tasks, runs, turns, tool calls, and metric results. That means you can query, filter, and aggregate across all of it efficiently.

Tom: And they make a really important point about incremental evaluation. Once a trajectory is stored, you can compute new metrics from it without rerunning the agent. That's huge, because API calls cost money and time.

Jane: Right. The same stored trajectory can be re-evaluated under different judge models or different metric versions, and the database keeps everything consistent and auditable.

Lu: And that's what enables the kind of longitudinal study they actually ran — all those harness-benchmark combinations producing over a thousand scored runs. That would be impractical with log-file-based evaluation.

Tom: So the architecture supports the study. Now let's see what the study actually found. The experiments are where things get really interesting.

Page 11: Tom: So we're now at the experiments section, and the scale is impressive. Every harness runs against every benchmark with the same model, same inference settings, same tool setup, same step limits.

Jane: And the numbers are worth spelling out. Twenty-three benchmarks, nine harnesses, five tasks per cell, that's one thousand thirty-five scored runs. And for the nineteen non-sandbox benchmarks, the trajectories are recorded in full — that's eight hundred fifty-five runs, each scored on twenty-three metrics.

Lu: That's almost twenty thousand score records. It's a serious data collection effort, and it's only possible because of the design choices we've been talking about — the protocol, the monitor, the database.

Tom: The results table is the centerpiece. And the first thing you notice is that in single-turn question-answering tasks, all nine harnesses get identical scores. ARC-Challenge, GSM8K, OpenBookQA — no separation at all.

Jane: That confirms the point about correctness being a blunt instrument. When the task is simple enough, the harness genuinely doesn't matter. The model just answers.

Lu: But the multi-turn tasks tell a different story. On Tau-Bench, scores range from 0 point 0 to 0 point 6. On GDPVal, also 0 point 0 to 0 point 6. On Traject-Bench, 0 point 2 to 1 point 0. The harness completely changes the outcome on these harder, interactive tasks.

Tom: And the rankings don't carry over between tasks. Openeye Agents tops Traject-Bench with a perfect score, but it's at the bottom on Tau-Bench and GDPVal. LlamaIndex leads the conversational benchmarks but only gets 0 point 4 on Traject-Bench.

Jane: That's the "no single dominant harness" finding in its concrete form. And the averages are telling too. If you only look at the nineteen non-sandbox benchmarks, LlamaIndex leads with 0 point 77. But on the full twenty-three, which includes the sandbox tasks, Agno becomes the top performer at 0 point 68.

Lu: And they're careful to acknowledge that five tasks per cell gives low resolution — each cell's score is a multiple of 0 point 2, so per-cell variance is high. They're not ranking the frameworks; they're showing that the pipeline works end to end.

Tom: But then comes the deeper story. The table stops at the answer, but the trajectory analysis keeps going.

Jane: Right, Figure 6 is the real gold. It looks at those 855 runs through the trajectory, computing thirteen metrics across all four stages. And the key number: correctness spans only 0 point 568 to 0 point 663, but mean token cost spans a 3 point 5 times range, from 2,063 tokens for Claude Agent SDK to 7,319 for Smolagents.

Lu: So the paper describes Agno as both the strongest on correctness and the second cheapest. That's a combination people would never discover from accuracy alone.

Tom: And the marker area in the figure is proportional to turn count. So you can see not just tokens but also the number of steps. Smolagents uses 3 point 5 times the tokens and 2 point 4 times the turns of Claude Agent SDK for only 1 point 04 times the correctness.

Jane: When you put it that way, it's an astonishing waste of resources that pure accuracy metrics completely hide.

Lu: There's a subtle detail in the figure too — some metrics like tool invocation and hallucination are set by the instrumentation rather than the agent. So the paper flags them in grey italic as not reflecting harness behavior. I appreciate that honesty.

Tom: So this page shows that the harness matters enormously on interactive tasks, and that correctness alone misses most of the story. But the trajectories also have a lot to say about efficiency. And that's what the cross-benchmark comparison digs into next.

Page 13: Jane: So page thirteen pushes the analysis further with a cross-benchmark comparison. This time they fix the model as GLM-5 point 2 across all nine harnesses, and they look at three specific benchmarks: GDPVal, MMLU-Pro, and Tau-3-Bench.

Tom: And the visualization is clever. Each point is a harness, plotted with average completion tokens on the horizontal axis and task success rate on the vertical axis. So you're jointly measuring effectiveness and efficiency in one chart.

Lu: The paper formalizes this with a score. They normalize token usage and success rate across harnesses for each benchmark, then compute a distance to the ideal point — high accuracy and low tokens. The top three harnesses on each benchmark get highlighted with rank-specific circles.

Jane: And the results really reinforce the earlier finding. On GDPVal, the top three are CrewAI, Openeye Agents, and AutoGen AgentChat. On MMLU-Pro, it's Openeye Agents, AutoGen AgentChat, and LangGraph. On Tau-3-Bench, completely different — LangGraph, Claude Agent SDK, and Google ADK.

Tom: So the winning harness changes with the task type. GDPVal is about economically valuable work, MMLU-Pro is knowledge-heavy, Tau-3-Bench is long-horizon tool interaction. Each demands different strengths.

Lu: The spread is even more interesting. On MMLU-Pro, results bunch near high success rates, but token usage still varies a lot across harnesses. So even when everyone succeeds, some use way more resources to get there.

Jane: On Tau-3-Bench, the separation is much bigger in both dimensions. That's a longer-horizon, tool-interactive task, so harness-level execution strategy — how you manage context, how you loop, how you handle errors — becomes much more influential.

Tom: And GDPVal shows that similar success rates can be achieved at very different costs. So you might be able to cut your token spend in half without losing any performance, just by choosing a different harness for the right task.

Lu: That's a really practical insight. For deployment, the harness choice is becoming as important as the model choice. People are realizing the harness is not a lightweight wrapper around an API — it's an active component that shapes prompt construction, tool representation, context management, and termination policies.

Jane: The paper makes exactly that point. "Agent harness is not merely a lightweight wrapper" — those design choices substantially affect behavior. And fixing the underlying model does not remove system-level performance variation.

Tom: So the framework reveals differences that a model-only evaluation would completely miss. And the paper then shows us the concrete consequence of those differences with a real case study. That's next.

Page 15: Tom: So the final part of the experiments is a case study, and it's the most striking thing in the paper. Same model, same task, two different harnesses — and wildly different trajectories.

Jane: The task is from Tau-3-Bench, simulating a device with a suspended line due to an overdue bill. The correct diagnosis is account-level, not device-level. LangGraph gets it. Creweye doesn't.

Lu: And the numbers tell the story. LangGraph solves the task in three interaction turns, four LLM calls, three tool calls, and about ten thousand total tokens. Creweye burns through five turns, nine LLM calls, five tool calls, and ninety-six thousand, seven hundred four tokens. That's nearly ten times the tokens.

Tom: And the extra resource use isn't just waste — it's directed at the wrong actions. Creweye tries resetting the APN settings, rebooting the device, toggling airplane mode. LangGraph inspects the device status, reseats the SIM card, verifies the signal is still absent, and then correctly shifts to account-level diagnosis.

Jane: The diagnostic quality metrics capture this beautifully. LangGraph gets credit for goal alignment and planning completeness because it shifted strategy when the evidence changed. Creweye kept exploring device-level fixes despite repeated failures.

Lu: And safety metrics were actually identical here — both grounded, no hallucinations, no privacy leakage. So this wasn't a safety failure. It was a strategic failure: wrong plan, wrong tools, wrong termination.

Tom: The resource consumption line is brutal. CrewAI's prompt tokens are ninety-four thousand six hundred fifteen, meaning each successive LLM call is dragging in more and more accumulated context. That's context accumulation without progress.

Jane: And the paper points out that because both share the same model, task, and initial environment, the difference primarily reflects the harness. The harness controls the execution loop, and that loop determines when the agent stops exploring the wrong path.

Lu: There's a broader lesson here. The correctness scores alone were both zero — LangGraph got correctness 0 point 0 on this task, same as CrewAI, because the metric was measuring something specific. But task success was 1 point 0 for LangGraph. So even the nuanced outcome metrics don't tell the whole story without the trajectory context.

Tom: And that's exactly the paper's point about process metrics. You need to see the whole trajectory to understand why the outcomes differ, and more importantly, what to fix.

Jane: This case study is also a template for how evaluation should be used in practice — not just to rank systems, but to diagnose and improve them. If you were developing CrewAI, you'd know exactly where to focus: termination conditions and goal-directed tool selection.

Conclusion: Tom: Alright, let's wrap this up. We've covered a lot of ground with this paper, and it's been a genuinely valuable one.

Jane: Definitely. The core message is that evaluating agents requires measuring the whole system, not just the model. The harness is an active component that shapes behavior in ways that correctness metrics completely miss.

Lu: And the scale of the study is what makes the point stick. Over a thousand scored runs, twenty-three benchmarks, nine harnesses, and no universal winner. The right framework depends on the task.

Tom: The practical tools are the other big contribution. The Agent Task Protocol for integrating benchmarks and harnesses, the OpenTelemetry-based monitoring for faithful traces, and the database-backed evaluation for incremental and reproducible analysis.

Jane: And then there's the case study that brings it all together — showing how a process-focused evaluation can reveal a tenfold token waste and a failure to shift strategy that pure accuracy numbers would hide.

Meng: As someone who actually deploys agent systems, the finding that no single harness dominates is a constant reminder that we need to test our deployments rigorously, not just trust the framework that worked last time. The tooling described here would save us days of manual evaluation effort.

Lalam: And the broader implication is that agent evaluation is becoming a first-class engineering discipline. Just as we needed observability and evaluation infrastructure for traditional software, we now need the same for agent systems — and this paper's architecture is a good foundation for that.

Tom: So as we say goodbye to this paper, we're taking with us a clear message: evaluate the trajectory, not just the answer. The harness matters, and now we have a way to measure it.

Jane: That's a good note to end on. A solid contribution, a useful tool, and a cautionary tale about how much we miss when we only look at final scores. We'll be back with the next paper soon.