You Only Align Once: Propagating Cooperative Behaviors in Multi-Agent Systems through Seed Agents

arXiv:2605.27586 · cs.MA, cs.CL · Submitted 2026-05-26 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "You Only Align Once".

Tom: A single aligned agent can propagate cooperative behaviors to untrained agents purely through natural language interaction, a phenomenon termed Alignment Propagation.

Jane: First, who's behind it and why it matters.

Title and authors: Tom: So, we're talking about "You Only Align Once: Propagating Cooperative Behaviors in Multi-Agent Systems through Seed Agents." The authors are Nicole Hsing, Asuka Yuxi Zheng, Yi Zhao, and Haoqin Tu. It really highlights the idea that alignment isn't something you fix piece by piece for every agent; instead, you create a single effective model to start the cooperation chain.

Jane: That’s right, Tom. The core title suggests we only need one aligned entity to spread those cooperative behaviors through natural language interaction. It reframes multi-agent alignment from a tedious per-agent training problem into something that can be engineered by strategically placing these initial seed agents.

Lu: What I find compelling about the authors' approach is how they specifically study this in the Red-Black Game, which is an iterated Prisoner’s Dilemma scenario, and then test its transferability to completely different environments like Sugarscape. That suggests a really deep understanding of what makes cooperative reasoning robust enough to survive shifts in narrative framing.

Meng: I'm curious about the specifics of how they distill the teacher model into that seed agent; the methodology sounds quite involved, involving several stages from input construction right down to LoRA fine-tuning. I want to know if this process is scalable or if it requires an enormous initial investment in high-quality data generation.

Lalam: If we can automate that distillation process, it moves us closer to creating AI that can self-optimize its social behaviors based on demonstrated collective success, which is a powerful cultural shift for any system we deploy.

The paper's summary: Tom: So, what does the paper actually show in terms of results? Basically, they demonstrate that taking the cooperative reasoning from a teacher model and creating a seed agent allows it to significantly boost cooperation among untrained teammates. For example, when placed among four untrained teammates in the Red-Black Game, this single seed agent doubled the cooperation rate from twenty-four point eight percent up to sixty-two point two percent <ref:2605.27586#pg0,when placed among four untrained teammates>.

Jane: That doubling effect is pretty substantial, Tom. It shows that simply having one persuasive agent can have a massive impact on the entire group's outcome through its dialogue and reasoning capabilities when interacting with others who haven't been trained in that specific cooperative framework yet.

Lu: The methodology involves distilling the teacher model’s cooperative dialogues into this seed using a process that enforces properties like "Situational Analysis" and "Collective Welfare Framing," which seems to be the key ingredient allowing it to generalize its persuasive ability across different game scenarios.

Meng: The data generation pipeline they used is quite detailed, including stages for input construction, teacher model selection based on performance metrics, ideal response generation using meta-prompts, and a quality control filter based on adherence to principles like "Principle Adherence" and "Influence Effectiveness." That level of rigor in filtering the training data is impressive.

Lalam: The paper emphasizes that this isn't just about one specific game; they tested it across different scenarios, including climate policy discussions and AGI safety narratives, suggesting the underlying persuasive mechanism is quite generalizable beyond the immediate game context.

The paper's improvements: Tom: Regarding improvements, the authors really focus on making this system more robust for real-world use by testing it across three different settings. They move from in-distribution scenarios like the Red-Black Game to scenario-level out-of-distribution tests, and even to a completely different environment like Sugarscape.

Jane: That shift from the familiar game structure to a spatially grounded survival simulation shows they are checking if this cooperative persuasion actually carries over when the underlying dynamics change significantly, which is a crucial test for any generalizable AI capability.

Lu: One key improvement they highlight is the demonstration of zero-shot transfer to Sugarscape, where the Qwen3-14B seed achieved a "ninety-one point five percent trade success rate versus a twenty-one point six percent baseline," showing strong performance in that unseen environment with no further task-specific fine-tuning needed.

Meng: That zero-shot transfer is what I find most practical for deployment; it means we can deploy an agent trained on one type of negotiation and expect it to function effectively in a completely different type of trading or resource competition environment without needing days of new training data for that specific task.

Lalam: The research also isolates two distinct channels: semantic persuasion in broadcast settings and dispositional consistency in pairwise settings, which gives us a much clearer picture of *how* the propagation actually happens, rather than just observing that it works.

Conclusion: Tom: So to wrap up on "You Only Align Once: Propagating Cooperative Behaviors in Multi-Agent Systems through Seed Agents," the main implication is that we can achieve high levels of cooperation in large agent populations by focusing on creating a single, persuasive seed that can spread its aligned reasoning through natural language.

Jane: Essentially, this points toward an engineering path where instead of trying to align every single member of a team individually, we focus our effort on perfecting the initial alignment mechanism so it can organically stabilize norms across the whole group.

Lu: The impact on the field is significant because it suggests that social AI capabilities can be engineered through distillation and strategic seeding, moving alignment from an exhaustive training problem to a scalable capability.

Meng: From an engineering standpoint, this means we might prioritize creating better teacher models and more precise quality control metrics for those initial seed agents, as the success seems highly dependent on the fidelity of that initial distillation step.

Lalam: I think the long-term impact is fostering systems that develop a positive moral trajectory through sustained interaction, which is a deep cultural improvement for how AI entities operate together.

Tom: Fantastic points from everyone. We've seen how this work moves us toward engineering scalable cooperative behaviors through seed agents in the paper "You Only Align Once: Propagating Cooperative Behaviors in Multi-Agent Systems through Seed Agents." It’s a lot to digest, but definitely something to keep an eye on as we look at future research on multi-agent systems.

Nicole Hsing, Asuka Yuxi Zheng, Yi Zhao, Haoqin Tu, Jen-tse Huang

University of California, Santa Cruz · Northwestern University · Johns Hopkins University

cs.MA, cs.CL

Submitted: 2026-05-26

Updated: 2026-10-02

Code: https://github.com/arcarae/YOAO

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: A single aligned agent can propagate cooperative behaviors to untrained agents purely through natural language interaction, a phenomenon termed Alignment Propagation.

Key concepts

Alignment Propagation
This is the core idea where one aligned agent can spread cooperative behaviors to others simply by talking to them. It turns multi-agent alignment from a hard training task for everyone into a scalable process based on strategically placing these trained 'seed' agents.
Seed Agent
A single AI model that has been specially fine-tuned (SFT) using high-quality cooperative dialogues from a superior 'teacher' model. This seed agent is then introduced to untrained teammates to start the alignment process, acting as a catalyst for cooperation.
Semantic Persuasion
This channel of propagation occurs in broadcast settings where aligned agents convince teammates by using reasoned arguments and principled dialogue. It means that simply having an aligned agent present isn't enough; they must articulate cooperative reasoning to shift teammate votes.
Dispositional Consistency
In pairwise environments, alignment happens through reliable behavior rather than direct persuasion. Agents develop a positive moral trajectory where consistent, mutually beneficial actions lead them to naturally cooperate and sustain good exchanges.

Terminology

Summary

A single aligned agent can propagate cooperative behaviors to untrained agents purely through natural language interaction, a phenomenon termed Alignment Propagation. This research reframes multi-agent alignment from an exhaustive per-agent training problem into a scalable social capability that can be engineered through strategic seed placement.

Alignment Propagation Mechanism

The core finding demonstrates that supervised fine-tuning (SFT) instills a robust, cooperative persuasion capability that generalizes across multi-agent interactions. This propagation is formally defined as Alignment Propagation. The process involves distilling the cooperative reasoning and persuasive dialogues of a high-performing teacher model into a single seed agent. This seed agent, when placed among untrained teammates, significantly increases the cooperation rate—for instance, doubling it from 24.8% to 62.2% in the Red-Black Game when placed among four untrained teammates.

Data Generation and Training Pipeline

The pipeline for creating these seed agents involves several distinct stages:

  1. Input Construction: Building context per turn consisting of a System Prompt (including agent identity, game rules, payoff matrix, and objective framing), Round Information (game state like round number and multiplier), and Prior Context (teammates’ messages).

  2. Teacher Model Selection: Evaluating seven LLMs against ten opponent strategies to select the generator for SFT training data and the base model for SFT. Kimi-K2-Thinking was selected as the generator because it achieved the highest average welfare (127/150) and had no negative scenarios.

  3. Ideal Response Generation: Using a meta-prompt that enforces five key properties: Situational Analysis, Social Awareness, Collective Welfare Framing, and Principled Robustness. This ensures responses analyze the game state, reference prior speakers, reason about collective outcomes, maintain cooperation after exploitation, and use persuasion through dialogue.

  4. Quality Control: Filtering data using a scalar reward based on four components: Principle Adherence (0.3), Collective Welfare (0.3), Influence Effectiveness (0.2), and Robustness (0.2). This process selects instances where agents both choose cooperation and articulate principled reasoning, excluding data characterized by purely outcome-oriented reasoning or retaliatory logic.

  5. LoRA SFT: Fine-tuning the selected base model (e.g., Qwen3-14B) using Low-Rank Adaptation (LoRA) with specific hyperparameters to introduce approximately 1.2B trainable parameters, transforming the worst-performing model into an effective alignment propagator.

Testing Across Diverse Environments

The paper evaluates Alignment Propagation across three settings with increasing distribution shift:

  1. In-distribution (ID): Red-Black Game scenarios used during training (e.g., Climate, Pandemic, AGI Safety).

  2. Scenario-level out-of-distribution (OOD): Heldout Red-Black Game scenarios that share game mechanics but introduce unseen narrative framings.

  3. Environment-level OOD: Sugarscape, a spatially grounded survival simulation with pairwise trading and fundamentally different dynamics compared to broadcast deliberation.

Key Results and Transferability

The results show that the aligned seed agents exhibit robust zero-shot transfer to the OOD Sugarscape environment. In this setting, the Qwen3-14B seed achieves a 91.5% trade success rate versus a 21.6% baseline. Furthermore, cross-architecture propagation is observed: Qwen-SFT seeds successfully influence the behavior of LLaMA-3.1-8B and Mistral-Small-3.1-24B, revealing a distinct persuadability spectrum across different architectures. The shift is attributed not to model capability alone but to the SFT's instilled capacity to actively persuade teammates and stabilize mutually beneficial norms.

Mechanisms of Propagation

The research isolates two distinct channels through which alignment propagates:

  1. Semantic Persuasion in Broadcast Settings: This channel involves trained agents shifting teammates’ votes through principled argument when they engage in deliberation. The study shows that semantic persuasion—not mere presence or action signaling—drives propagation.

  2. Dispositional Consistency in Pairwise Settings: In environments like Sugarscape, aligned agents succeed not by convincing partners but by behaving reliably enough to sustain mutually beneficial exchange. This is linked to the development of a positive moral trajectory, where agents with more successful trades develop cooperation and shift toward cooperation (∆l > 0).

Scaling and Topology

The efficiency of propagation scales positively with group size. The analysis shows that for teams of N=5, the tipping point is around 40% seeds; however, for N≥10, "just 20% seeds yield 98–100% cooperation with zero variance.

Improvements for AI systems

Based on the scientific paper You Only Align Once: Propagating Cooperative Behaviors in Multi-Agent Systems through Seed Agents, here are specific, actionable improvements for AI systems and what those improved systems can achieve:

) 1. Implement a Scalable Alignment Propagation Pipeline (The Core Improvement)

The primary improvement is shifting the paradigm from exhaustive per-agent training to a seed placement strategy. This involves developing an automated pipeline that can distill cooperative reasoning from powerful foundation models and propagate it across an entire population with minimal additional training cost.

  1. Specific System Improvements:

Adaptive Seed Generation & Distillation:

Create a robust, multi-stage data generation and distillation pipeline (Stages 1-5) that systematically extracts high-quality, principled cooperative reasoning traces from top-performing teacher models (e.g., Kimi-K2). This pipeline must be rigorously filtered using quality control metrics (Influence Effectiveness and Robustness) to ensure the resulting seed agent possesses genuine persuasive capacity, not just reactive compliance.

  1. Capabilities of the Improved System:

Automated Alignment Injection:

The improved system can take a single, highly distilled seed agent and strategically inject it into a large population of untrained or unaligned agents (e.g., in a multi-agent simulation or real-world deployment). This seed agent will use natural language interaction to persuade its neighbors, effectively doubling the cooperation rate from baseline levels (e.g., 24.8% to 62.2% in a Red-Black Game) purely through dialogue and argumentation, without requiring the entire population to be retrained.

  1. Specific System Improvements:

Environment-Agnostic Transfer Learning:

The distillation process must be designed not just for one game (Red-Black Game), but for Scenario Prompts that capture the underlying mechanics of cooperative decision-making (e.g., collective welfare maximization, principled reasoning). The resulting seed agent should then demonstrate robust zero-shot transfer to fundamentally different environments with distinct dynamics, such as spatial survival simulations like Sugarscape.

  1. Capabilities of the Improved System:

Zero-Shot Cooperative Adaptation:

The improved AI system can deploy a single seed agent trained on one domain (e.g., negotiation) and have it immediately function effectively in a completely new, unseen context (e.g., pairwise trading/resource competition), achieving high success rates without any further task-specific fine-tuning.

  1. Specific System Improvements:

Topology-Aware Propagation Control:

The system should incorporate a mechanism to determine the optimal seed ratio based on the agent group size and interaction topology (broadcast vs. pairwise). For broadcast settings, it can recommend a minimal seed fraction (e.g., 20%) for maximum efficiency, while for decentralized settings, it can suggest higher coverage ratios (e.g., 50%).

  1. Capabilities of the Improved System:

Optimized Resource Allocation:

In resource-constrained multi-agent environments (like GPU clusters), the system can use learned cooperative patterns to dynamically adjust resource allocation requests (Standard vs. Priority) to maximize total collective throughput, mitigating contention and maximizing shared success across teams.

  1. Specific System Improvements:

Norm Internalization Monitoring:

The system should include a monitoring layer that tracks Norm Persistence after the initial seeds are removed. This involves measuring long-term shifts in agent identity leaning (e.g., from self-interest to cooperation) based on their trade success history, allowing for dynamic adjustment of seed density or further targeted intervention if norms collapse.

  1. Capabilities of the Improved System:

Resilient Collective Behavior:

The improved system can foster stable, mutually beneficial norms that resist degradation under adversarial prompts or when faced with conflicting local incentives (e.g., exploitation). It moves beyond simple compliance to instill a dispositional consistency that actively reshapes agent beliefs through sustained interaction.

Abstract

Ensuring aligned agent behaviors in distributed open multi-agent systems remains challenging, especially as populations grow and unaligned agents may exist. We show that a single aligned agent can propagate cooperative behaviors to unmodified agents purely through natural-language interaction, a phenomenon we term Alignment Propagation. We study this in the Red-Black Game, a team-based iterated Prisoner's Dilemma in which teammates deliberate and vote to determine their team's collective action. By distilling the cooperative reasoning and persuasive dialogues of a teacher model into Qwen3-14B, we obtain a seed agent that, when placed among four unmodified teammates, more than doubles the cooperation rate from 24.8% to 62.2%, outperforming the teacher model and a vanilla Gemini-3.1-Pro. Remarkably, a seed trained exclusively on the Red-Black Game transfers zero-shot to Sugarscape, a spatially grounded survival simulation with pairwise trading, achieving a 91.5% trade success rate versus a 21.6% baseline. Our results reframe multi-agent alignment from an exhaustive per-agent training problem to a scalable social capability that can be engineered through strategic seed placement.

Sources

Related papers