What Survives Self-Evolution: Provenance Marks in Copied Agent Harnesses

summary

Video file (mp4)

The gist

The gist The first measurement of provenance marks under self-evolution shows that a key in the order of near-tied entries in a table built from public games survives covert rewriting attacks by an

In short

The research tested how provenance marks survive covert rewriting attacks when an AI agent harness is evolved by a plagiarist. The key found was the specific order of near-tied entries in a table derived from public games, which resists deletion and rewrites. Survival depends on selection mechanisms weighing the carrier's value only after sufficient game decisions are made.

Key concepts

Covert Rewriting Attack
This is when an LLM proposer modifies a file carrying a provenance mark without targeting the mark itself. The goal is to rewrite the file while maintaining or improving its score, making it difficult for owners to detect that the deployed agent descends from the original package.
INSTANCE_ORDER Mark
The key used to track survival is 'INSTANCE_ORDER,' which represents a specific sequence of finding and placing numbered objects in a table. This order is derived from public games, making it hard for attackers to delete or reorder without breaking the mark.
Selection Mechanism
Survival depends on how decisions are made during evolution. A mark only protects itself if the selection process observes enough games. If a decision doesn't see enough games, the protection fails, allowing rewrites that change the order.

Terminology used across episodes

This episode discusses

The paper

What Survives Self-Evolution: Provenance Marks in Copied Agent Harnesses · Read on arXiv

Jieyu Li, Jingya Li, Sheng Li, Xinpeng Zhang, Zhenxing Qian

Fudan University

Agent harnesses, the prompt and tool code around a base model, leak. Claude Code's 512,000 lines were mirrored within hours of a packaging error. A plagiarist can pass a leaked harness through a published self-evolver and ship the result as its own. We define this as a covert rewriting attack. No one targets a provenance mark, yet a large language model (LLM) proposer rewrites the file that carries it and a score keeps the better rewrites. We give the first measurement of marks under this attack, with three evolvers, five proposers and two environments. Published marks fail. KGW in the prompt survives GEPA's rewrites in 0 of 12 copies, AgentMark as released in 5 of 12. The proposer's habits decide survival, by two orders of magnitude across models, and selection protects a mark only when each decision sees enough games. We then build a mark for this attack. Its key is the order of near-tied entries of a table of where objects are found, built from public games, so deleting the table costs success, rewrites that keep its rows keep the order, and a customer reads the key from the agent's actions alone. On held-out games it is detected in 62 of 76 copies whose lineage rewrote its file. A scheme-aware plagiarist reshuffles the ties at no measurable cost, and a plagiarist who asks the proposer for more general code removes the key in 75-100% of rewrites.

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: I'm Nadia.

Elias: And I'm Elias, here with Priya.

Nadia: Today's paper: "What Survives Self-Evolution".

Elias: The gist The first measurement of provenance marks under self-evolution shows that a key in the order of near-tied entries in a table built from public games survives covert rewriting…

Nadia: First, who's behind it and why it matters.

Paper summary: Nadia: So looking at the whole paper, "What Survives Self-Evolution: Provenance Marks in Copied Agent Harnesses," it seems the main finding is that a key placed in the order of near-tied entries within a table built from public games survives covert rewriting attacks <ref:2610.21458#pg2>.

Elias: And this survival happens provided that selection weighs the carrier's value only when each decision sees enough games <ref:2610.21458#pg3>.

Priya: It boils down to this: if you structure your data so that deleting the table costs success, and rewrites keep its rows in order, then a customer can read the key from the agent’s actions alone <ref:2610.21458#pg2>.

Nadia: So for someone just listening to this show, what does that mean? It means that a specific way of organizing information can give it enough resistance against people trying to sneakily modify it through self-evolution <ref:2610.21458#pg3>.

Conclusion: Nadia: So we're wrapping up this deep dive into "What Survives Self-Evolution: Provenance Marks in Copied Agent Harnesses." The core idea is that they found a way for a specific piece of data—a provenance mark—to survive when an AI proposer keeps rewriting the files it carries over time >

Elias: Exactly. They focused on how to make this mark resilient against those rewrites, suggesting the key isn't just about hiding it, but structuring the data around it so that deleting it actually hurts the process of making a better rewrite >

Priya: It seems like they’re saying if you build your data in a way that makes losing that specific order impossible without consequence, then even if someone tries to completely overhaul the agent harness, this tiny piece of information can stick around >

Nadia: Right. It means we aren't just looking for perfect copies; we’re looking for structures that are inherently resistant to being overwritten by an evolving AI system >

Elias: And what this changes for us is understanding how selection works in these high-stakes rewriting games. It points toward a defense strategy where the value of a piece of data isn't static; it depends on how many times it gets tested across different models >

Priya: It makes me wonder if this kind of structural protection could apply to more than just agent harnesses, maybe to other critical system components that need to maintain a traceable lineage through constant modification >

More episodes

← Home