EMAS: Stabilizing Multi-Agent System Evolution through Evidence-Guided Revision

arXiv:2608.07196 · cs.AI · Submitted 2026-08-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "EMAS: Stabilizing Multi-Agent System Evolution through Evidence-Guided Revision".

Jane: The paper was written by Chao Fei, Qingyi Si, Kaihua Liang, Yanghua Xiao, Panos Kalnis et al. from King Abdullah University of Science and Technology and JD.com and Fudan University.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary: Tom: So we have a brand new preprint in front of us, and it tackles a question that's been hanging over the whole multi-agent field — can these systems actually get better as they work, not just after some initial design phase?

Jane: And the answer the authors give is yes, but with real discipline attached. The method is called EMAS, and the core idea is that a multi-agent system should revise its own prompts and topology using evidence gathered from actual task executions.

Tom: The system is represented as a graph, where each node is a single small step — one LLM call — and edges carry artifacts between steps. Every execution leaves a trace, and the traces get turned into structured diagnoses. Something like the verifier step never receiving the solver's output.

Lu: What I find most distinctive is the gating. A single bad sample never triggers anything. The same diagnosis has to show up across multiple distinct samples before the system even considers a revision. Then the proposed revision gets validated head-to-head against the current system on a fixed validation set.

Meng: So you need both recurrence and empirical proof before anything changes. That's what "evidence-guided revision" means in practice — it sharply reduces the risk of chasing noise.

Lalam: And it shifts the whole framing from designing a clever system to authorizing changes. The system becomes something that accumulates auditable state over time, rather than a static configuration frozen the moment deployment starts.

Jane: The numbers are genuinely strong. Across four benchmarks and two frozen backbones, evolved systems reach the highest overall accuracy in six of eight model–benchmark settings. The most dramatic case is code generation with Qwen, where accuracy climbs from about 55 percent to 89 percent while token use per task drops by over 62 percent.

Lu: And the ablations cut the other way too — take away the validation gate and the same setup collapses from around 95 percent accuracy down to about 66 percent on Game24. The safeguards are doing real work.

Meng: The paper is also honest about the fact that individual revisions can regress. Roughly 40 percent of committed revisions hurt the test score. It's the overall trajectory that wins.

Lalam: That honesty matters because it tells us what's actually hard. And the first page of the paper lays that out directly — the authors frame three specific questions about where to revise, when to revise, and how to keep regressions under control.

Tom: That's exactly where we should go next — page one, and the motivation behind the whole framework.

Page 1 of the paper: Tom: So picking up where we left off, page one is the introduction, and it starts with the idea of recursive self-improvement — systems that improve the reusable mechanisms shaping their future behavior.

Jane: And the key move is what counts as "mechanisms." For an LLM-based multi-agent system, it's not just weights. Prompts assign responsibilities, computation decomposes work, and topology routes intermediate artifacts. That whole layer around the model is something you can evolve.

Lu: Right, so they're saying the executable system layer is mutable even when the model weights are frozen. That's the whole foundation — evolution at the system level, not the parameter level.

Tom: And then they pose three questions that structure the entire paper. Where do you revise? When do you revise? And how do you control regressions across revisions?

Meng: The "where" question is interesting because a wrong answer can have so many causes. The paper lists missing computation, harmful information flow, and inadequate instructions as distinct failure classes. You need a representation that exposes intermediate computation to tell them apart.

Jane: That's why they build on GPTSwarm's graph view — an MAS as a graph of steps connected by directed edges. It makes the internal computation observable, which is a precondition for any localized diagnosis.

Tom: And the "when" question is about distinguishing isolated failures from recurring patterns. That's the recurrence gate we mentioned. The "how" question is handled by paired validation against the current system.

Lu: They also introduce the two evolution directions up front — improving accuracy, and reducing token use without sacrificing accuracy. Both are treated as legitimate objectives.

Meng: The validation set being balanced with equal numbers of correct and incorrect samples is a detail that struck me. It prevents the validation from being dominated by easy cases.

Jane: That balanced setup matters because a revision could just overfit to whatever distribution the validation happens to have. Balancing it forces the comparison to be informative on both failure repair and regression detection.

Lalam: So page one is really setting up a contrast — model training changes weights through backpropagation, while EMAS changes the system around the model based on execution experience. Same learning spirit, totally different object being updated.

Tom: And that contrast gets fleshed out with a figure on the next page, so that's where we're headed.

Page 2 of the paper: Tom: So page two opens with that figure we just mentioned, and it's the clearest picture of what's different here.

Jane: On one side you have classic model training — a mini-batch, a loss, backpropagation, updated weights. On the other side you have EMAS — the current system runs tasks, produces traces and feedback, and that evidence supports a discrete candidate revision.

Lu: And the key part of the figure is the loop. If the candidate passes validation, it becomes the next version of the system. If it doesn't, you keep the current one. That's a state transition, not a gradient step.

Tom: The page also states the headline results pretty early — 6 point 30 percent relative gain in overall accuracy for Kimi, 20 point 10 percent for Qwen, within just two evolution epochs.

Meng: Those relative numbers look different partly because of starting points. Qwen starts much weaker, so it has more room to climb. Kimi starts strong and improves more modestly.

Jane: Then there's the extended Game24 result. Going to fifteen epochs, Kimi reaches 96 point 45 percent accuracy and Qwen reaches 94 point 73 percent — and Qwen's tokens per task drop by almost 47 percent along the way.

Lalam: I like that they report both the two-epoch results and the long-horizon results. It shows the evolution keeps paying off rather than plateauing immediately.

Lu: The page also previews the ablation numbers — without the validation gate, a committed transition is 1 point 57 times as likely to regress, and the average accuracy loss per regressive transition is about 18 times larger. Those are

Page 3 of the paper: Tom: So we've seen the big picture — EMAS keeps the model frozen but lets the system around it evolve — and page three zooms out to show how this fits with everything else that's been tried.

Jane: It's the related work section, but instead of just listing papers, the authors organize the field around three decisions you have to make every time you change a system. What state can change, what evidence justifies a change, and what comparison lets you commit the change.

Tom: The first bucket is about what's mutable. Prior work has treated prompts, tools, graph connections, and whole agent workflows as learnable system state, all above the frozen model weights.

Jane: That means EMAS isn't claiming the idea of editable systems is new. GPTSwarm already represented agents as graphs, and ADAS and AFlow already searched over complete workflows. The novelty is in how you authorize changes, not in the fact that you can make them.

Tom: The second bucket is about evidence, and this is where it gets interesting. A wrong answer can mean several different things — missing computation, bad information flow, or weak instructions.

Jane: So the paper brings in work like Trace, which uses execution traces as feedback, and semantic backpropagation, which pushes natural-language feedback through the agent graph. Both stress that this is genuinely hard, which is why automated failure attribution is its own

Page 4 of the paper: Tom: So we just saw how EMAS positions itself against prior work, and now page four starts laying out the actual machinery—how the system is represented and where the evolution loop begins.

Jane: Right, and the first thing they do is make the representation really explicit. An MAS becomes a graph where every node is a single atomic step, meaning one focused LLM call, and edges carry the intermediate artifacts between steps.

Tom: That granularity is what separates this from treating an agent as one big block. They decompose an agent's responsibility into multiple steps, so if something goes wrong, you can point to which exact step or connection caused it.

Jane: And they go one step further by building a separate MAS for each task category. So math problems don't share a system with planning tasks. Each category gets its own graph and its own prompts, which makes diagnosis cleaner.

Tom: The page also introduces the version concept. A version is just a particular state of the graph and prompts, and you only advance to a new version when a revision is accepted. Rejected candidates leave you where you were.

Jane: That sounds safe, but the real safety comes from the two checkpoints. First, a hypothesis has to recur across multiple distinct samples before it even becomes a candidate. Then the candidate has to beat the current version in paired validation.

Tom: And there's a nice detail about how the initial system is built. The designer looks at a small set of representative tasks and generates the whole topology and prompts, then a structural validation pass checks for broken edges and missing prompt coverage.

Jane: So the starting point is a perfectly valid but possibly imperfect system. That fits the whole philosophy—you don't need a brilliant first design, you need one that can be diagnosed and evolved.

Tom: The page also mentions two objectives, accuracy and cost. Wrong answers produce accuracy-oriented diagnoses, but correct answers can generate cost hypotheses, like spotting a redundant edge or a step that just adds tokens.

Jane: That dual track is important because it means evolution doesn't stop once accuracy saturates. Instead, it shifts toward making the system cheaper to run.

Tom: Now, the actual diagnosis engine—how a trace becomes a structured revision hypothesis—that's the very next page, and it's where the real design choices start showing.

Page 5 of the paper: Tom: We've seen how EMAS represents a multi-agent system as a graph of atomic steps, and page five zooms into how that representation gets used for diagnosis and repair.

Jane: Right, and the first detail is that this graph is a directed acyclic graph. Each step is one focused LLM call, edges carry artifacts forward, and prompts sit at several levels – system, category, phase, and step. That structure makes failures addressable instead of being buried inside a black box.

Tom: They even evolve a different MAS for each task category within a benchmark. So algebra gets one graph, counting and probability gets another. That granularity keeps the diagnosis honest because you're not mixing unrelated failure patterns.

Jane: Then comes the initialization. The designer looks at a handful of representative tasks, produces the whole topology and prompts, and a structural validation pass repairs any broken edge or missing prompt coverage. It's not meant to be perfect – just valid enough to start evolving.

Tom: The real engine is online evolve. Each sample runs through the current system, and the trace gets converted into a structured diagnosis. Every diagnosis records the objective, the defect class, the operation, and the exact location.

Jane: So a wrong answer might produce a hypothesis like "this step lacks an edge from the solver," and that hypothesis points precisely at where a fix would go. That's far more surgical than saying "this agent failed."

Tom: The operations are equally fine-grained. You can add or remove a node, split a node into more atomic pieces, add or remove an edge, or edit a single prompt. They give a clean example: if a verifier never receives the solver's output, that becomes an add-edge hypothesis at a canonical location.

Jane: Every revision is a small, bounded change. Nothing rewrites the whole system on a hunch. Next comes the part that decides whether a hypothesis gets acted on at all – the recurrence gate and then the validation test.

Tom: Let's get to that.

Page 6 of the paper: Tom: So we've seen how a trace becomes a structured hypothesis, and now page six shows the two gates that decide whether that hypothesis ever becomes a real change to the system.

Jane: Exactly, and the first gate is recurrence with operation-specific thresholds. The paper makes a practical point: adding a step or edge is safer because it just adds redundancy, but removing something can break what other samples depend on, so removals need more evidence.

Tom: They set different thresholds for that, and you need more supporting samples before you're allowed to remove a node or edge than to add one. The logic is that destructive edits carry more risk, so they should wait for stronger confirmation.

Jane: Then there's the scope rule. Each candidate is one primary change, and you can only touch directly affected edges and prompts. Adding a node doesn't give you license to rewrite the whole graph.

Tom: That bounded scope is what keeps evolution auditable. You can look at any version and know exactly what changed and why, instead of a mysterious blob of modifications.

Jane: Right, and then comes the second gate: paired validation. The candidate and the current system run on the same fixed validation set, and the acceptance rules are strict. An accuracy fix must strictly increase the correct count, and a cost fix must keep or improve accuracy while strictly lowering tokens.

Tom: If the candidate fails, you just keep the current version. The rejected candidate gets recorded, but it has no effect on what runs next.

Jane: There's also a clever detail about stale evidence. After an accepted revision, they re-execute the remaining samples under the new system, so old diagnoses don't trigger proposals that were already addressed.

Tom: That replay step stops the system from acting on outdated complaints. It's the kind of thing that sounds obvious but is easy to miss in practice.

Jane: Now the methodology is complete, and the next page starts the experiments section, where they put this whole machinery to the test across benchmarks and backbones.

Page 7 of the paper: Tom: So we've covered how EMAS only commits a change after repeated evidence and paired validation, and page seven finally shows what that discipline buys you in practice.

Jane: This is the main results table, and it's dense. Every cell reports accuracy plus tokens per task for both backbones across Math, MBPP, PlanBench, and Game24, comparing EMAS against the initial design, a few baselines, and two prior automated design methods.

Tom: The first thing that jumps out is the overall numbers. Kimi goes from 90 point 11 percent to 95 point 79 percent task-weighted accuracy, and Qwen goes from 73 point 42 to 88 point 18 percent. Those are relative gains of about 6 and 20 percent.

Jane: The gap makes sense because Qwen starts with a much weaker initial MAS, so there's more headroom to repair. Kimi is already near saturation, so its gains are smaller but still real.

Tom: And the comparison against Genesis plus SOP is telling. That baseline takes the same standard operating procedure but keeps it as a single prompt instead of decomposing it into a graph. The Initial MAS beats it, which shows the structure itself is doing work, not just the text of the SOP.

Jane: EMAS also wins on task-weighted accuracy against AFlow and ADAS with both backbones, and it's best or tied in six of the eight individual settings. That's a strong showing.

Tom: The most dramatic cell is Qwen on MBPP. Accuracy climbs from 55 point 09 percent to 89 point 12 percent, and tokens per task drop from 5 point 16k to 1 point 95k. That's a 62 percent cost reduction alongside a massive accuracy jump.

Jane: But the paper is also honest that token changes are mixed across other settings. Under just two epochs, evolution mostly focuses on accuracy, and not every category reaches a cheaper state. That would probably require more evolution time.

Tom: So page seven gives us the headline results. The obvious next question is how these improvements actually unfold across versions, and whether they keep going beyond two epochs.

Page 8 of the paper: Tom: We just covered the headline results on page seven, and now page eight shows how those gains actually unfold over time with the evolution trajectories.

Jane: Right, Figure 3 is the heart of this page. Each panel shows a category's accuracy and token cost across committed versions, and the overall pattern is clear – accuracy goes up, tokens go down, especially in the long-horizon Game24 runs.

Tom: But what I find more interesting is that the character of evolution changes as a system matures. Early on, with lots of headroom, you see big accuracy jumps. Once a category gets near saturation, accuracy moves in a narrow band and the evolution starts focusing on cutting token cost.

Jane: That matches the two objectives we discussed. The system naturally shifts from fixing errors to trimming waste, and that happens without anyone explicitly switching modes.

Tom: Then there's an honest admission. Individual revisions can actually hurt on the test set, and later versions have to recover and surpass earlier states. They give two reasons: validation acceptance doesn't guarantee test improvement, and the vLLM backend can produce small numerical instabilities.

Jane: So they use a checkpoint mechanism that keeps track of the best-performing version. That's a practical safeguard, not a theoretical guarantee.

Tom: The extended Game24 evolution makes the backbone difference visible. Kimi gets to high accuracy quickly and then makes small refinements. Qwen needs a much longer repair phase, but the eventual gain is bigger – it goes from 79 point 9 to 94 point 7 percent, while also cutting tokens per task nearly in half.

Jane: That tells you the same procedure adapts to the capability of each model. It keeps expanding the accuracy-cost frontier instead of plateauing.

Tom: Now, these trajectories assume both safeguards are working. The next page tests that assumption directly by ablating recurrence and validation gating.

Jane: Let's hear it.

Page 9 of the paper: Tom: So we've seen the evolution trajectories, and now page nine puts the two safeguards under the microscope to see what actually breaks when you remove them.

Jane: Right, the ablation on Game24 with Qwen sets up three configurations. Full EMAS with both recurrence and validation, then a version where a single trace triggers a revision, and then a version with no validation gate at all.

Tom: The gap is dramatic. Full EMAS reaches 94 point 7 percent accuracy with 3 point 48k tokens per task. Without the validation gate, the best accuracy collapses to 65 point 5 percent, and the average regression per transition jumps from less than a third of a percent to nearly 8 points.

Jane: It's the validation gate that stops destructive simplification. Removing it doesn't even save tokens — you get 4 point 26k tokens per task, which is actually higher than full EMAS. The gate isn't slowing things down; it's keeping bad changes out.

Tom: Recurrence plays a different role. With a single trace, you get eight regressive transitions instead of seven, but the cumulative loss more than doubles, from 5 point 59 points to 13 point 92. So recurrence isn't just avoiding an occasional hiccup — it's reducing how much noise can distort the whole trajectory.

Jane: The paper then wraps up with a very honest conclusion. Evolution is non-monotonic. On held-out test data, 39 of 93 committed revisions actually regress, and the headline results use the best checkpoints found retrospectively.

Tom: That's a real limitation. The checkpoints show those states are reachable, but there's no deployment-time rule for knowing which one you're in. You can't peek at the test set while the system is running.

Jane: They also note that recurrence doesn't mean causal equivalence. Two traces can share the same structured key but have genuinely different causes, and a single bounded edit might not be enough to fix something that needs coordinated changes.

Tom: And the cost of paired validation is substantial. Every candidate runs over the same validation set, which adds up quickly. They're honest that this hasn't been tested beyond two backbones and four benchmarks.

Jane: Still, the closing line is powerful — a frozen model need not imply a frozen executor. That's the whole philosophy in one sentence.

Tom: So that's where the main text ends, but the appendix actually carries a lot of the gritty detail about thresholds and provenance.

Jane: Let's dig into that next.

Page 10 of the paper: Tom: So we've walked through the entire method and results, and page ten closes with two short but meaningful statements about eye use and reproducibility.

Jane: Right, and the eye use statement is refreshingly specific. They say generative tools helped with editing code, LaTeX, figures, tables, and readability, but every assisted piece was reviewed by the authors and numerical results were checked against underlying records.

Tom: That level of disclosure is becoming more common, but the reproducibility statement goes a step further. They mention a recorded result bundle, provenance manifests, checksums, and scripts that regenerate all the tables and figures.

Jane: The checks actually reject incomplete records, inconsistent aggregates, unregistered artifacts, and failed numerical invariants. So the paper isn't just asking you to trust the numbers — it's giving you tools to verify them.

Tom: And that's meaningful because this is a paper about systems that change themselves. If you're going to let an eye system evolve, you need to be able to audit exactly what changed and why, otherwise the whole thing becomes a black box.

Jane: The reproducibility statement fits the philosophy of EMAS itself. Every revision leaves a trace, every version is a documented state, and the audit trail is part of the design, not an afterthought.

Tom: There is one thing that struck me, though. These statements appear at the end, but they're quite brief. The real weight of the paper is in the appendix tables, which we haven't fully explored.

Jane: That's true. The appendix has the support thresholds, the full breakdown of which revisions were accepted and rejected, and the evolution cost in tokens.

Tom: So let's pull back the curtain on that appendix material, especially the thresholds and the cost analysis.

Jane: Let's do it.

Conclusion: Tom: We've walked through EMAS from the motivation to the appendix, and here's the whole thing in one sentence: EMAS treats a multi-agent system as a mutable graph of atomic steps and only changes it when recurring evidence, validated head-to-head against the current version, says the change is worth keeping.

Jane: That framing really turns self-improvement into an authorization problem. You're not just generating edits; you're deciding which edits earn the right to shape future behavior.

Tom: And the results speak for themselves. Higher accuracy on both backbones, in most settings, with the standout being Qwen on MBPP climbing from 55 to 89 percent while cutting tokens by 62 percent.

Jane: But the ablations are what convinced me. Removing the validation gate drops accuracy from nearly 95 percent to 65 percent. Those safeguards aren't overhead; they're what makes evolution stable enough to trust.

Tom: The authors also deserve credit for being transparent about the rough edges. About 40 percent of committed revisions regress on test, and the headline results use the best checkpoints found retrospectively, not something you could pick at deployment time.

Jane: They're also clear about the cost. Paired validation is expensive, the fixed validation set can cause adaptive selection, and the current structured diagnosis might merge causes that are actually distinct.

Tom: Still, the big picture is exciting. The model weights stay frozen, yet the system around them learns and accumulates auditable state. That's a meaningful step toward recursive self-improvement that we can actually inspect.

Jane: And the future work list is just as interesting: rolling validation, retention-aware acceptance, coordinated revisions, cheaper screening. There's a clear roadmap here.

Tom: So this one gets a solid mark from us. It's a well-executed idea with honest limitations and a practical direction forward.

Jane: And next up on the show, we've got a paper that tackles a completely different angle on agent reliability, so stay tuned.

Chao Fei, Qingyi Si, Kaihua Liang, Yanghua Xiao, Panos Kalnis, Hongcheng Guo

King Abdullah University of Science and Technology · JD.com · Fudan University

cs.AI

Submitted: 2026-08-07

Updated: 2026-08-10

Code: https://github.com/cf3i/Evolving-Multi-Agent-System

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 79/100

The gist: The paper introduces EMAS (Evolving Multi-Agent System), a method for evolving LLM-based multi-agent systems (MAS) at the system level—revising prompts and topology while keeping the underlying LLM

Key concepts

Multi-agent system (MAS): A system where multiple AI agents, each with specific roles, collaborate to solve tasks. In EMAS, it's represented as a graph of steps (LLM calls) with edges carrying data, allowing fine-grained diagnosis and revision.
Evidence-guided revision: A process where changes to the system are only made after recurring evidence from multiple task executions supports a specific diagnosis, and the proposed change passes a head-to-head validation against the current system on a fixed set.
Recurrence gate: A safety mechanism requiring the same diagnosis to appear across multiple distinct samples before a revision is considered. This prevents acting on isolated failures, with stricter thresholds for destructive changes like removing nodes or edges.
Paired validation: A test where the candidate revised system and the current system run on the same validation set. Acceptance requires strict improvement in accuracy or cost without regression, ensuring only beneficial changes are committed.

Terminology

Accuracy — 160× in this paper · explained in 1 episode
Benchmark — 40× in this paper · explained in 2 episodes
Trace — 29× in this paper · explained in 2 episodes
Ablation — 13× in this paper · explained in 1 episode
Multi-Agent System — 13× in this paper · explained in 6 episodes
Agentic System — 4× in this paper · explained in 2 episodes
Evolving Multi-Agent System — 4× in this paper
Frozen model — 3× in this paper · explained in 3 episodes
Adaptive Selection — 2× in this paper · explained in 1 episode
Provenance — 2× in this paper · explained in 4 episodes
Reinforcement Learning — 2× in this paper · explained in 4 episodes
Verifier — 2× in this paper · explained in 2 episodes
Ablation study — 1× in this paper · explained in 1 episode
Adam — 1× in this paper · explained in 2 episodes
Agentic Workflows — 1× in this paper · explained in 1 episode
Aggregation — 1× in this paper · explained in 1 episode
Failure Modes — 1× in this paper · explained in 3 episodes
Generative AI — 1× in this paper · explained in 2 episodes
Multi-agent system (MAS) — 1× in this paper · explained in 2 episodes
Redundancy — 1× in this paper · explained in 2 episodes
Self-Evolving Agents — 1× in this paper · explained in 2 episodes
Topological Order — 1× in this paper · explained in 2 episodes

Summary

The paper introduces EMAS (Evolving Multi-Agent System), a method for evolving LLM-based multi-agent systems (MAS) at the system level—revising prompts and topology while keeping the underlying LLM frozen. The authors note that "Many methods for automated multi-agent system design optimize prompts and topologies during an initial design stage and then deploy the resulting system unchanged on subsequent samples. Experience from these samples is rarely consolidated into reusable system updates, while accuracy-oriented designs may incur high token costs."

EMAS addresses this by us[ing] this experience to revise MAS topology and prompts without updating LLM parameters, either to improve accuracy or to reduce cost. The method "converts traces into structured diagnoses that specify a revision operation and target. It generates a candidate revision only when the same diagnosis recurs across samples and applies it only if paired validation against the current MAS meets the corresponding acceptance criterion."

The paper frames evolution around three questions: where to revise the system, when to revise it, and how to control regressions across revisions. A wrong answer may result from missing computation, harmful information flow, or inadequate instructions, and a revision should either improve accuracy or reduce token use without lowering accuracy.

Building on GPTSwarm's graph view, EMAS represents an MAS as a graph of Steps connected by directed edges. Each Step performs one fine-grained LLM call, and the edges carry intermediate artifacts between Steps. Each node is defined as an atomic operation, thereby decomposing an Agent's responsibility across multiple nodes. To enable targeted optimization, EMAS constructs and evolves a separate MAS for each task category.

The MAS state for category c at version v is defined as M c(v) = (G c(v), P c(v)), where "G c(v) specifies the graph's LLM-call nodes and edges, and P c(v) specifies the prompt for each node. A version therefore denotes a particular MAS state. Only an accepted revision advances v; a rejected candidate leaves the current version unchanged."

The evolution lifecycle contains two checkpoints: "Step-level traces first produce localized, structured revision hypotheses. EMAS generates a candidate only when the same hypothesis recurs across enough distinct Evolve samples... The candidate is then compared with the current version on the same Validation set."

Diagnosis. Each trace yields records canonicalized as h = (o, d, a, l), where o ∈ ACCURACY, COST denotes the objective, d denotes the defect class, a denotes the associated operation, and l denotes its canonical location. Correct outputs produce cost-oriented hypotheses searching for avoidable computation, information transfer, or instructions while preserving the correct outcome.

Revision operations. EMAS can modify Steps, Edges, and prompts: "ADD-NODE addresses a missing operation, REMOVE-NODE addresses a redundant or harmful operation, and SPLIT-NODE addresses a non-atomic operation. For Edges, ADD-EDGE supplies missing information to a Step, whereas REMOVE-EDGE eliminates redundant or harmful artifact flow. For prompts, PROMPT-ONLY revises the relevant prompt to provide better instructions."

Recurrence gating. "For each hypothesis h, EMAS maintains an evidence buffer containing the distinct samples whose traces support h. Candidate construction is triggered only when the number of supporting samples reaches the operation-specific threshold κ a. Operation-specific thresholds reflect risk: Adding a Step or Edge is generally less risky... By contrast, removal is riskier because it can eliminate computation or information flow required by some samples." Thresholds range from 4 (ADD-NODE, SPLIT-NODE, ADD-EDGE) to 6 (PROMPT-ONLY) to 10 (REMOVE-EDGE, REMOVE-NODE).

Paired Validation. The candidate and current MAS are evaluated on the same fixed Validation set. Acceptance rules are:

Accuracy: An accuracy-oriented candidate is accepted only if it strictly increases the aggregate correct count.
Cost: A cost-oriented candidate is accepted only if it maintains or increases the aggregate correct count while strictly reducing the total execution token cost.

On rejection, EMAS retains the current MAS; on acceptance, the candidate is promoted. Following acceptance, EMAS re-executes the remaining active samples under the new MAS to obtain up-to-date diagnoses. This replay prevents stale evidence from triggering subsequent candidate proposals.

EMAS was evaluated on four benchmarks (Math, MBPP, PlanBench, Game24) with two frozen LLM backbones (Kimi-K2-6 and Qwen3.6-27B).

Main results (two epochs): EMAS attains the highest task-weighted overall accuracy for both backbones and is best or tied in six of eight model–benchmark settings. Specifically:

Kimi-K2-6: overall accuracy rises from 90.11% to 95.79%... corresponding to relative gains of 6.30%
Qwen3.6-27B: overall accuracy rises from 73.42% to 88.18%... corresponding to relative gains of 20.10%
Evolution improves strong initial systems across models and tasks... Relative to the Initial MAS, accuracy improves in seven settings.
On Qwen–MBPP: EMAS raises accuracy from 55.09% to 89.12% while reducing tokens per task from 5.16k to 1.95k, a 62.2% reduction.

The paper notes that "Evolution gains depend on the headroom and maturity of the MAS. Qwen starts from a weaker Initial MAS and realizes a larger relative gain... Kimi starts from a stronger Initial MAS, and its near-saturated Math setting leaves less room for accuracy improvement."

The Genesis+SOP baseline—which adds the SOP text to the Genesis prompt without decomposing it into an MAS—has lower task-weighted accuracy than Initial MAS with both backbones, indicating that the executable MAS structure contributes beyond the SOP text alone.

Extended evolution: Extending Game24 to Epoch 15, the best Kimi checkpoint improves from 95.34% accuracy and 12,971 tokens per task to 96.45% and 12,585 tokens, while Qwen improves from 79.90% and 6,539 tokens to 94.73% and 3,481 tokens (a 46.77% token reduction for Qwen). Thus, the same evolution procedure adapts to the capability of each backbone and can continue to expand the reachable accuracy–cost frontier beyond the main two-epoch budget.

Evolution dynamics: Individual revisions can move accuracy or token use in the wrong direction on the test set, after which later Versions may recover and surpass earlier states. This occurs because acceptance on Validation does not guarantee improvement on the Test set and because small numerical instabilities can occasionally produce different answers. EMAS therefore uses a checkpoint mechanism to retain the best-performing Version. The paper observes that accuracy-directed evolution is more prominent in the earlier stages, whereas cost-directed evolution becomes more prominent once correctness approaches saturation.

Selectivity of revisions: Across both backbones, 255 of 1,118 triggers are accepted (22.8%). Kimi accepts 157 of 559 triggers (28.1%); Qwen accepts 98 of 559 (17.5%). Looking at 93 E2 revisions with exact-predecessor Test comparisons, 39 revisions reduce Test accuracy, with an average loss of 3.03 percentage points. Successful accuracy-directed revisions gain 4.46 points on average. Additionally, 28 of 59 cost-directed revisions reduce tokens without lowering accuracy, saving 427 tokens per task on average.

Ablations (Game24 with Qwen3.6-27B): The paper tests removing evidence accumulation (κ = 1) and removing the Validation gate. "Relative to Full EMAS, a committed transition is 1.14 times as likely to regress under the single-trace variant and 1.57 times as likely without the Validation gate. The average accuracy loss per regressive transition is also 2.18 times as large under the single-trace variant and 18.05 times as large without the Validation gate. Full EMAS achieves 94.73% accuracy and 3.48k tokens; κ=1 achieves 89.58% and 12.07k tokens; no gate achieves 65.56% and 4.26k tokens. Recurrence improves the evidence used to identify a system-level defect, whereas Validation tests the quality of the concrete candidate proposed for that defect."

The paper lists three contributions:

We formulate system-level evolution for MASs, where experience from subsequent tasks updates prompts and topology while the LLM weights remain fixed.
We introduce EMAS, which turns recurring task experience into validated system updates for improving accuracy and reducing token use.
Across four benchmarks and two frozen LLMs, we show that EMAS improves accuracy and can reduce token use, while recurrence and Validation make evolution more stable.

The authors acknowledge several limitations: "Evolution nevertheless remains non-monotonic. Repeated use of a fixed Validation set can induce adaptive selection, aggregate improvement need not preserve every solved task, and 39 of 93 committed E2 revisions with exact-predecessor comparisons regress on Test. The headline results use the best checkpoints observed retrospectively on Test. These checkpoints show that better states are reachable, but they do not provide a deployment-time rule for identifying them. Furthermore, Recurrence also does not imply causal equivalence because the current structured key can merge traces with distinct causes, while a single bounded edit may miss repairs that require coordinated changes. EMAS also assumes correctness feedback, incurs substantial paired-Validation cost, and has been evaluated on only two backbones and four benchmarks. Future work should explore rolling validation, retention-aware acceptance, coordinated revisions, and cheaper screening. The paper concludes: A frozen model need not imply a frozen executor, but its evolution must generalize, preserve prior capabilities, and pay for itself."

Improvements for AI systems

Here are specific, implementable improvements to AI systems based on EMAS, and what the improved system can do.

Add system-level self-evolution from runtime traces

Instead of deploying a fixed multi-agent system, the AI system continuously converts execution traces into structured diagnoses of the form (objective, defect, operation, location). This enables the system to identify where it failed or wasted tokens, and to propose targeted updates to prompts, graph topology, or node composition.

Use recurrence gating before proposing any revision

Maintain an evidence buffer per diagnosis and trigger a candidate revision only when the same diagnosis recurs across a threshold number of distinct samples (e.g., 4 for additions, 6 for prompt changes, 10 for removals). This prevents the system from overreacting to single outliers or noisy traces.

Validate every candidate against the current system on the same validation set

The system only accepts a revision if it strictly improves aggregate accuracy, or if it reduces total token cost without lowering accuracy. This paired-validation gate controls regressions and ensures every accepted change is accountable.

Support atomic, targeted revision operations on the agent graph

Represent the system as a graph of fine-grained steps with prompts and edges. Allow operations such as ADD-NODE, REMOVE-NODE, SPLIT-NODE, ADD-EDGE, REMOVE-EDGE, and PROMPT-ONLY. This lets the system add missing computation, remove redundant/harmful information flow, split overloaded nodes, or improve instructions with minimal, precise edits.

Separate and evolve a dedicated system per task category

The system maintains a distinct MAS version for each task category, so revisions are localized to the actual failing behavior rather than globally changing all behavior.

Replay remaining samples after an accepted revision

After promoting a new version, the system re-executes still-active samples to produce fresh diagnoses. This prevents stale evidence from triggering inappropriate follow-up revisions.

Keep the best-performing checkpoint despite non-monotonic evolution

Since individual revisions can temporarily hurt test performance, the system retains the best observed version and can roll back or compare checkpoints, making the overall evolution more robust.

Run accuracy-directed evolution early, cost-directed evolution later

The system can schedule or prioritize revisions based on saturation: first improve correctness, then shift to cost reduction once accuracy plateaus. This expands the accuracy–cost frontier over time.

Freeze LLM weights while updating only prompts and topology

The system improves without fine-tuning or retraining the underlying model, making it cheap, backward-compatible, and applicable to black-box or API-only LLMs.

Explicitly separate improve accuracy and reduce cost objectives

The system can propose cost-oriented hypotheses from correct traces (e.g., removable steps, redundant edges, unnecessary instructions) and accuracy-oriented hypotheses from incorrect traces, with different acceptance criteria for each.

Self-improve after deployment: It learns from each task sample's execution trace and revises its own agent structure, prompts, and information flow without human intervention.
Improve accuracy significantly: In the paper's evaluations, it raised task-weighted accuracy by 6.3% on a strong backbone and 20.1% on a weaker backbone, including improving one benchmark from 55% to 89% accuracy.
Cut token costs while maintaining accuracy: It can reduce tokens per task by over 60% in some settings by removing redundant computation and unnecessary information flow, and by 46.8% in extended evolution tests.
Avoid harmful regressions: Recurrence thresholds and paired validation make committed revisions much less likely to degrade performance—reducing the average accuracy loss per regressive transition by over 2× compared to single-trace proposals and 18× without validation.
Adapt to the capability of the underlying model: The same evolution procedure automatically finds more aggressive cost reductions for weaker backbones and more accuracy-focused revisions for stronger backbones.
Continue improving beyond the initial design stage: The system keeps evolving over many epochs, reaching better accuracy–cost trade-offs than any fixed initial design.
Explain and localize its own failures: It outputs structured, operation-specific diagnoses (e.g., missing node at location X, redundant edge from Y to Z) rather than opaque global retraining signals.
Operate under frozen-model constraints: It works with black-box LLMs where parameter updates are impossible, enabling continuous improvement of deployed systems.