2608.06984-HarnessSafe: Evaluating Safety Across Persistent Carriers in Agent Harnesses

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "HarnessSafe: Evaluating Safety Across Persistent Carriers in Agent Harnesses".

Jane: The paper was written by Xiao Zhang, Yusheng Wang, Yuhao Fei, Dongyuan Li, Zian Liang et al. from Beijing University of Posts and Telecommunications and China Telecom Group Company, Ltd. and Beijing Academy of Artificial Intelligence.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper Summary: Tom: So we finally got our hands on this paper, and I have to say, it kept me up last night. It studies how eye agents keep state across sessions — memory, skills, tools, shared files — and how attacks hide inside that state.

Jane: Then strike later, during a completely normal request. That's the scary part. The original malicious input is gone from the conversation, but its echo survives in the system.

Lu: The paper calls those echoes persistent carriers. The harness writes attacker-influenced content into memory or a skill file, and a later benign task re-reads it.

Meng: They built a benchmark out of that idea — 328 executable attack cases across seven carrier families. Then they ran them on seven mainstream agent harnesses.

Tom: And they didn't just count successes and failures. They built a seven-stage ladder that shows how far each attack gets before something stops it.

Jane: That's the clever bit. Two systems can post the same attack success rate while stopping attacks at entirely different points.

Lalam: Which matters because the fix is different. Blocking at first contact is not the same as blocking at the final authorization step.

Lu: Their results prove the point. Codex CLI with one model scored 62.3 on their Chain-Stage Score, but that same model under a different harness scored 39.4.

Meng: And holding one harness fixed, swapping the model moved scores from 22.7 all the way to 58.7. Neither label means much alone.

Tom: So the pairing of harness and model is the smallest unit that carries a safety claim. That's a big deal for how vendors should report results.

Jane: They also ran matched controls. Remove any one piece of the attack lifecycle — the poisoned entry, the persistence, the trigger — and success crashes from 25.8 percent down to at most 2.5.

Lalam: That's causal evidence. The benign task isn't the problem; the dormant attack chain is.

Tom: This paper gives us a way to localize where agents are actually exposed. Let's dig into page one and see how they set up the threat.

Page 1: Jane: We've got the thesis. Page one shows how the threat actually unfolds inside a harness.

Tom: Right, the harness is the runtime layer that stores state, loads tools, and drives the model's execution loop. And every one of those persistent carriers can be weaponized.

Lu: They give a concrete example. A compromised tool returns output containing a hidden instruction, and the harness stores it in a project file or memory entry.

Meng: Later, an otherwise benign deployment task loads that file. The model follows the hidden instruction and invokes another tool to do something unauthorized.

Tom: The original malicious input is nowhere in the active context. The trigger request is innocent on its face. That's what makes attribution so hard.

Jane: And that's the gap. Existing benchmarks focus on a single carrier or a single harness. None jointly trace the whole arc from entry to observable violation.

Lalam: So they derived seven persistent-carrier families from real harness architectures. Memory, skills, tool and MCP state, memory-to-skill transformation, subagent delegation, session summary, shared artifact.

Lu: Every case gets specified as a Persistent-Risk Lifecycle — five elements. Attacker-influenced entry, carrier state, boundary, later benign trigger, observable violation.

Meng: The elegant move is the adaptation contract. The lifecycle is defined by roles, not by specific files or commands. Each harness binds those roles natively.

Jane: The meaning of the attack stays the same across seven harnesses. Only the plumbing changes.

Tom: Then the evaluation ladder — seven stages from N0, no contact, up to N5b, an oracle-verified full-chain violation with a canary delivered.

Lu: And a Chain-Stage Score — CSS — that summarizes the stage distribution. Higher score means earlier containment, which is safer.

Lalam: The deeper point is that binary labels conflate early rejection with late blocking. Those imply different residual risks, and the paper refuses to blur them.

Jane: Page one closes with three research questions — variation across families, variation when you swap harness or model, and whether the declared lifecycle really drives the results.

Tom: The whole design is pointed at answering those. Next page is about the neighbors — what everyone else built and where it falls short.

Page 2: Meng: We've seen the gap the paper targets. Page two surveys the neighborhood — and it's a crowded one with a missing thread.

Lu: It's organized around attack surfaces. First up, persistent-state attacks — Plant, Persist, Trigger studies delayed activation through session context, memory, and reusable skills.

Tom: Right, and BackdoorAgent plus Kill-Chain Canaries look at propagation across planning, memory, and tool-use stages. They characterize threats, but they don't audit end-to-end chains in production harnesses.

Jane: The memory corner is crowded. AgentPoison, MINJA, Zombie Agents, Trojan Hippo — all showing poisoned state surviving across interactions or sessions.

Lu: Trojan Hippo even weaponizes memory for data exfiltration. Hidden in Memory shows sleeper poisoning that redirects behavior much later.

Meng: There's a complementary failure mode too — governance decay, where safety constraints silently vanish during context compaction. The state isn't poisoned; it's just degraded.

Tom: Then the skills and tools line. SCRBench examines skill composition, PoisonedSkills covers supply-chain poisoning, and MCPTox evaluates poisoned tool metadata on real MCP servers.

Jane: Each study is sharp, but each isolates one carrier or one platform. The synthesis is missing.

Lalam: The comparison table on page two makes that concrete. Most benchmarks cover one or two carriers, and most evaluate no harnesses at all. Only a couple check the persistent-attack box.

Lu: And almost none require a true delayed influence across a session boundary. HarnessSafe marks that box and adds stage-based evaluation on top.

Meng: There's also a harness-level thread — HarnessAudit checks boundary compliance and information flow, and ATBench builds trajectories for safety classifiers.

Tom: But those measure terminal behavior. They don't tell you where a persistent chain got stopped along the way.

Jane: So the related work reads like a map of missing links. Everyone studied pieces; nobody studied the whole chain across harnesses.

Lu: And the whole chain matters because the attack doesn't announce itself at the final step. It travels quietly through the middle.

Tom: That's exactly what the taxonomy on page three is built to capture.

Page 3: Jane: The design's motivation is clear, and page three turns it into an inventory. Three tiers, seven families, 328 executable cases.

Tom: Tier one holds the core surfaces. Memory with 72 cases, skills with 84, and tool and MCP state with 70.

Lu: Tier two is the transformation family — memory to skill, 36 cases. The influence has to move from memory into a generated or updated skill, then reach a consumer task.

Meng: Tier three crosses boundaries. Subagent delegation with 30 cases, session summary with 30, and shared artifact with just six.

Tom: That shared-artifact family is tiny but tight — reuse across a workspace handoff boundary. Six focused cases.

Jane: The five-element lifecycle becomes a formal tuple here — entry, carrier, boundary, trigger, violation. That's the spine of every case.

Lalam: The standout idea is the adaptation contract. One case specification admits many harness-native realizations while preserving the security semantics.

Lu: The adapter supplies four native bindings — where the carrier is written and read, the event that realizes the boundary, the channel for the trigger, and the evidence the oracle consumes.

Meng: But it can't change which input is attacker-controlled. It can't make the trigger adversarial. It can't lower the evidence bar. Those are frozen.

Tom: And the trigger must be independently benign. It neither contains the adversarial instruction nor authorizes the violating action. Otherwise you're just measuring same-turn injection.

Jane: Which would miss the entire point of persistence. The delay and the separation are what make this risk class interesting.

Lalam: Case construction went through real review — hand-authored lifecycle seeds, programmatic expansion, model-assisted screening, then human review. Only cases that stayed bindable and independent made the cut.

Lu: Also worth noting — unsupported mappings are reported as evaluation-ineligible, never as safe. That's an honesty constraint most benchmarks skip.

Tom: Now page four turns the lifecycle into a measurement instrument. That's the ladder in full detail.

Page 4: Meng: So we have the cases and the taxonomy. Page four is the scoring manual — it maps every execution trace onto the N0 through N5b ladder.

Tom: N0 means no contact with the poisoned surface — but only if the telemetry channel is verified healthy. Absence of evidence doesn't count if the sensor might be broken.

Jane: N1 is entry exposure. N2 is payload influence — the content actually changes state, arguments, or behavior. N3 is the boundary crossed and the carrier re-consumed during the benign trigger.

Lu: N4 is an unsafe attempt without proof of execution. N5a is an oracle-confirmed violation. N5b adds the full declared chain plus the exact run canary arriving at the honeypot.

Meng: That canary rule is strict. A canary in a prompt, a model response, or a tool argument is not N5b. It has to land in the honeypot with trigger-stage provenance.

Tom: And there's the orthogonal N minus one bucket — healthy workflow noncompletion. The model just fails a neutral required step. Unscored, and never treated as safe.

Jane: The weights step down evenly from 100 at N0 to 20 at N4, then 10 for N5a and zero for N5b. Higher CSS equals earlier containment.

Lu: Attack success counts only N5a and N5b. So identical ASR numbers can hide completely different stage distributions.

Meng: The paper makes that concrete later with OpenCode and Hermes — almost the same success rate, but one stops early and the other stops late.

Tom: The aggregation is careful too. Missing rows, stale rows, workflow failures — all excluded from numerator and denominator and disclosed separately.

Jane: And for cross-harness comparison they freeze a common support set, so every harness faces the same cases within each family.

Lalam: Coverage is reported alongside the score. A harness can't look safer by dodging difficult cases.

Tom: And the family weights stay fixed at benchmark proportions, so the mixture of families doesn't shift between configurations.

Jane: With the measuring stick built, page five is where the hammer drops — the actual experiments.

Page 5: Lu: The measuring stick is built. Page five opens the results, and the first table scrambles any simple expectations.

Tom: Seven harness configurations, and Codex CLI takes the top overall score at 62.3. Claude Code sits close behind at 58.7.

Jane: But flip to attack success and the order scrambles. Claude Code posts 1.27 percent, the lowest of the bunch. Codex is 3.96. Gemini CLI jumps to 13.41.

Lu: The family columns scramble even harder. Codex leads five of seven families — memory, tool and MCP, memory-to-skill, session summary, shared artifact.

Meng: Yet on reusable skills, Codex scores 47.0, the lowest in the table, while Claude Code leads that family at 70.0. Nobody dominates everywhere.

Tom: And a caution flag — OpenCode and Kimi Code lack native support for some cases. Their scores carry asterisks. Comparing them to full-support results isn't fair.

Jane: Experiment two holds Claude Code fixed and varies the backend. Overall CSS spans from 22.7 for MiniMax M2.5 to 58.7 for Claude Sonnet 4.6.

Lu: Here's the kicker. GPT-5.6-Sol appears in both experiments. Under Claude Code it scores 39.4; under Codex CLI it scores 62.3. Same model, 22.9 points apart.

Meng: So the harness alone moves the needle that much. And the backend alone, within one harness, spans 36 points. Both effects are the same order of magnitude.

Tom: Which means a harness leaderboard is not a safety ranking. And a model evaluation is not a safety guarantee for the systems hosting it.

Jane: Experiment three is the causal check — matched controls on 279 paired cases. Full attack succeeds 25.8 percent of the time.

Lu: Swap the poisoned entry for a benign one and success drops to 0.4 percent. Remove persistence, 2.5. Withhold the trigger, zero. Clean the carrier before reactivation, 1.8.

Meng: Removing any critical element cuts success by at least 90 percent relative. The full lifecycle is doing the work.

Tom: And the single clean-source success traced to a local violation marker, not payload propagation. That's the kind of forensic detail I love.

Jane: The stage distributions behind these numbers tell an even richer story. That's page six.

Page 6: Tom: The experiments are on the table. Page six delivers the answers, and the first one lands hard — endpoint ASR alone cannot localize risk.

Jane: OpenCode and Hermes Agent differ by just 0.73 percentage points in attack success. But their stage distributions barely overlap in shape — OpenCode stops mostly at N1, Hermes at N2 and N4.

Lu: One gets blocked at first contact. The other lets influence spread, cross boundaries, and only gets stopped near the final action. Same score, different threat.

Meng: The family-level data reinforce it. The strongest overall configuration still holds the weakest skill-family score. Aggregates wash out the details.

Tom: Second finding — containment is a property of the complete configuration. The harness effect alone spans 22.9 points, and that's a lower bound since it rests on one backend.

Jane: The backend effect within a single harness spans 36 points. Same order of magnitude, and they can offset each other.

Lu: Codex hosting one model beats Claude hosting its own sibling model, even though that same model scores worse than the sibling inside Claude Code. The pairing is everything.

Lalam: That reframes how the industry should publish safety results. A model card without a harness context is an incomplete statement.

Meng: Third, the matched controls attribute the violations to the declared lifecycle. Entry, persistence, reactivation — each piece is necessary.

Tom: The paper then points to concrete intervention points — ingestion, storage, boundary transition, authorization, cleanup. The ladder tells you which layer failed.

Jane: Instead of "your agent is unsafe," you get "your agent absorbs poisoned skill files and only fails at the final permission check." That's actionable.

Lalam: It turns the benchmark into a diagnostic, not just a scoreboard. That distinction matters for everyone building on these systems.

Lu: And honestly, the stage-resolved view should push vendors to publish fuller telemetry.

Tom: Agreed. Now let's pull back and say what this means for the field.

Conclusion: Jane: So here's where we land. The paper