One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL

The paper addresses a critical failure mode in multi-agent reinforcement learning for human-AI interaction. Multi-agent reinforcement learning for human-AI interaction typically relies on a single large language model to simulate user behavior. The authors show that this approach systematically fails to generalize, and trace the failure to simulator collapse: because the simulator LLM is mode-collapsed, an LLM policy trained against it overfits to narrow strategies that exploit the simulator's dominant mode, and such a policy transfers poorly to unseen simulators and real users. They formalize this collapse theoretically and propose two complementary solutions, one at inference time and one at training time. The inference-time solution, Verbalized Sampling, broadens the simulator's behavior by sampling from a verbalized response distribution, reducing mode collapse. The training-time solution, Co-Training, jointly optimizes the policy against a population of trainable simulators, preventing it from overfitting to any single simulator's mode. They validate both solutions on three multi-turn benchmarks: Persuasion for Good, τ2-bench, and CooperBench. Verbalized Sampling improves held-out success by up to 9% over single-simulator RL, and Co-Training pushes gains further to 14%; the human study shows similar gain on real users. Both solutions preserve the policy diversity that collapses under single-simulator RL. To support further work in this direction, they release SCOPE, an open-source framework for Population Co-Training multi-agent RL. More broadly, their results suggest that the diversity of the training environment, not only the policy, is critical to the generalization of multi-turn RL to real-world deployment.

Reinforcement learning with verifiable rewards has driven rapid progress on single-turn LLM capabilities, from mathematical reasoning to tool use to software engineering. Toward more realistic tasks beyond single-agent verifiable rewards, a growing line of work studies multi-turn human-AI interaction: customer support, collaborative coding, persuasion, and tutoring. But training RL with real users at scale is prohibitively expensive and slow, so prior work has turned to LLM-based user simulators.

A common method in recent work is to prompt a single frozen LLM to play the user. In this work, the authors show this practice can systematically fail to generalize. In multi-turn RL, the policy no longer interacts with a deterministic verifier; the simulated user becomes part of the training environment, and its output distribution decides which states the policy visits and what gradient signals it receives. They identify a systematic failure mode of this recipe, which they call simulator collapse. Because many aligned LLM simulators are mode-collapsed, a policy trained against a single frozen simulator receives gradients dominated by that simulator's modal behavior, and overfits to narrow strategies that exploit that mode. The error compounds across dialogue turns, so a policy trained this way fails when transferred to unseen simulators, and is unlikely to transfer to real users.

The main contributions are summarized as follows:

1. Identifying simulator collapse. They formalize how a mode-collapsed user simulator biases the policy gradient toward its mode and collapses the policy's entropy onto a narrow simulator-specific exploit. This defines a structural failure mode of RL against a single frozen simulator (§3.2).

2. Two solutions at different points in the training loop. Verbalized Sampling is the inference-time solution: at each simulator turn during rollout, the simulator is queried for a verbalized response distribution and one response is sampled from it, restoring within-simulator diversity without retraining either side. Co-Training is the training-time solution: they jointly update the policy and a trainable user simulator on the same rollouts, so the partner adapts as the policy improves. They release SCOPE, an open-source framework that unifies multi-model rotation, self-play, and dual-model Co-Training behind a single pluggable interface (§3.4).

3. Empirical validation across three multi-agent RL settings. On Persuasion for Good, τ2-bench, and CooperBench, single-simulator RL drops back toward the untrained baseline. Both Verbalized Sampling and Co-Training close most of the held-out gap, and Population Co-Training yields the strongest held-out task success rate (§4). They additionally run a human study on τ2-bench and Persuasion for Good, where Co-Training improves task outcome and both methods improve P4G dialogue naturalness over single-simulator RL (Appendix E).

The paper models multi-turn dialogue as a two-player partially observable Markov decision process (POMDP) with a shared conversation-history state. At each turn t, the state st is the full conversation history. The agent samples its utterance aπt ∼ πθ(· st); the user simulator then samples a response aϕt ∼ ϕψ(· st, aπt). A trajectory τ has terminal reward R(τ), and the agent maximizes J(θ; ψ) = Eτ∼(πθ,ϕψ)[R(τ)].

The state-visitation distribution is jointly determined by the agent and the simulator: the simulator does not only score trajectories, it determines which histories the policy learns from. They apply REINFORCE to full multi-turn trajectories using group-relative reward normalization. For each task they sample a group of G trajectories, score them by terminal reward, and form z-scored advantages assigned uniformly to all agent tokens in each trajectory.

Aligned LLMs favor typical responses under direct prompting; recent user-simulation studies confirm the same pattern, with LLM simulators reading as overly cooperative and stylistically uniform. The authors sharpen this into a definition tied to the policy's training rollouts: at the simulator turns the policy actually visits, the simulator's response distribution is mode-collapsed.

Definition 3.1 (Simulator collapse on the training rollouts). For a policy πθ and a threshold ϵ⋆ ∈ [0, 1], the simulator is ϵ⋆-collapsed on the training rollouts if E(st,aπt)∼(πθ,ϕψ) ϵϕ(st, aπt) ≤ ϵ⋆ at the simulator turns visited by rollouts from (πθ, ϕψ).

The hypothesis: A mode-collapsed simulator breaks multi-agent RL into a fixed-user setting: the RL policy learns the strategy that wins against the simulator's mode, and that strategy fails when other models or real users deviate from it.

Theorem 3.2 (Simulator collapse induces mode-user optimization). Assume rewards are bounded in [0, Rmax] and the trajectory-level policy score satisfies ∥Σt ∇θ log πθ(aπt st)∥ ≤ B. Couple πθ in Mϕ and Mmode turn by turn. Define the accumulated collapse error along the rollout as ϵ̄H(θ) = E[ΣHt=1 ϵϕ(st, aπt)]. Then DTV(Pϕθ, Pmodeθ) ≤ ϵ̄H(θ), and ‖∇θ Jϕ(θ) − ∇θ Jmode(θ)‖ ≤ 2BRmax ϵ̄H(θ).

Theorem 3.2 shows that the gradient does not vanish; it is biased, up to ϵ̄H(θ), toward the objective in which every user emits the mode a⋆ϕ.

Lemma 3.3 (Collapse removes simulator-side reward contrast). If the simulator's trajectory is ϵH(x, ξπ)-close in TV to the mode trajectory at (x, ξπ), then VarξU(Rx x, ξπ) ≤ R2max ϵH(x, ξπ). When simulator-side variance vanishes, the z-scored advantage measures only agent-side variation. Group-relative RL then ranks samples by how well they exploit the simulator's mode; user-robustness drops out of the comparison.

Proposition 3.4 (Mode advantage concentrates policy mass). Under a KL-regularized softmax policy-gradient step on qk(y x) with learning rate η and bounded Q-estimation errors, the log-odds of Ax satisfy log[qk+1(Axx)/qk+1(Acxx)] ≥ log[qk(Axx)/qk(Acxx)] + gx, where gx > 0 depends on η, the gap ∆x, and the simulator-collapse and estimation errors.

Corollary 3.5 (Entropy concentration). Iterating gives qk(Ax x) ≥ 1 / (1 + [1−q0(Axx)]/[q0(Axx)] e−kgx), so the strategy distribution concentrates onto Ax geometrically fast in k.

Proposition 3.6 (Deployment regret from missing user behaviors). Let Bx be a set of real-user behaviors requiring a strategy outside Ax, with real-user mass q⋆(x) = P⋆(Bx x). Suppose every ym ∈ Ax is worse than an adaptive strategy yb by at least ∆realx on Bx and by at most νx outside Bx, and the trained policy places probability at least 1 − αx on Ax. Then J⋆(yb x) − J⋆(π̂ x) ≥ (1 − αx)[∆realx q⋆(x) − νx(1 − q⋆(x))] − αx Rmax.

The authors examine these results empirically on Persuasion for Good and τ2-bench. They train against three frozen simulators of varying modal concentration (GPT-5-mini, Haiku-4.5, Gemini-3-Flash). Training reward climbs in every run, fastest for the most modal simulator. OOD eval peaks early and turns over, but the magnitudes differ sharply: the most modal simulator (Gemini-3-Flash) crashes below the untrained baseline, while the least modal (GPT-5-mini) sees a gentler but still clear decline. Policy entropy crashes toward zero across all three runs. The failure is on the policy side: the policy learns the narrow exploit and fails to transfer to unseen simulators.

The collapse chain rests on two load-bearing assumptions. First, the per-turn collapse error ϵϕ(st, aπt) stays close to zero across the horizon. Second, the modal-exploit set Ax stays fixed across training. The two fixes attack one assumption each.

Verbalized Sampling recovers the reference simulator distribution. A greedy query drives ϵϕ(s) toward zero. A verbalized query returns K candidate responses with verbalized probabilities; the resulting distribution pVSϕ(· s) approximates the simulator's pre-RLHF reference distribution P(· s), since the distribution-level prompt verbalizes the response distribution that direct aligned prompting sharpens away.

Proposition 3.7 (Reference-gradient recovery under Verbalized Sampling). If DTV(pVSϕ(· s, aπ), P(· s, aπ)) ≤ η(s, aπ) at every visited state along an H-turn rollout, then the trajectory distributions and policy gradients in MVS vs. the reference-user environment Mref satisfy DTV(PVSθ, Prefθ) ≤ η̄H(θ) and ‖∇θ JVS(θ) − ∇θ Jref(θ)‖ ≤ 2BRmax η̄H(θ), where η̄H(θ) = E[ΣHt=1 η(st, aπt)].

Co-Training lets the simulator and policy co-evolve. They update the user simulator on its own turns of the same conversation, so both sides receive gradients from a single rollout. The simulator's mode at each history shifts as training proceeds, and a strategy that exploited yesterday's mode no longer wins against today's: Ax is no longer constant, and the geometric concentration in Corollary 3.5 no longer applies.

Three multi-turn benchmarks: Persuasion for Good (P4G), a persuader arguing for a charitable donation against a resistant donor (reward r = min(donation/2, 1)); τ2-bench, customer-service dialogues on retail or airline tasks with binary per-split success; and CooperBench, two coding agents coordinating on a multi-step task with binary success. P4G uses its donation-based adversarial reward and CooperBench its symmetric task-success reward; on τ2-bench the simulator is trained with a SPICE-style curriculum reward that targets within-batch variance σ2 ≈ 0.25.

Six paradigms under matched setup: Base (untrained); RL (Single) against GPT-5-mini; Verbalized Sampling; Ensemble Models, cycling K=3 frozen models from different families; Co-Training, pairing the policy with a separately trained simulator; and Population Co-Training (ours), sampling the active simulator from a pool of recent checkpoints. Plus a Persona-Guided simulator, which conditions a single GPT-5-mini simulator on per-rollout personas.

Q1: Does single-simulator RL collapse? Yes, but the collapse is in the curve shape, not the headline number. The best checkpoint is a transient peak that collapses back toward Base on τ2-Retail before training ends. At end of training, RL (Single) and Persona-Guided sit within a few points of Base on both τ2 splits. The population methods stay at or above their peak throughout.

Q2: Do the two proposed solutions recover the signal? Both do. On Qwen3-4B-Instruct, Verbalized Sampling lifts τ2-Retail from 46.1 to 55.5, Airline from 29.8 to 36.9, and P4G reward from 0.275 to 0.484. Co-Training lifts Retail and Airline further to 60.5 and 44.4. Population Co-Training tops both at 62.2 and 45.7 and is the best method on every τ2 split at both model sizes and on P4G at 4B.

Symmetric cooperation. On CooperBench, frozen-partner cross-play plateaus at a ceiling set by the partner's capacity; only self-play and population self-play break through, with the pool variant reaching the same peak faster.

Q4: Do the gains transfer to real users? They run a human study on τ2-bench (retail split) and Persuasion for Good with four conditions per task: Base, RL (Single), +Verbalized Sampling, and +Co-Training, collecting N=40 Prolific participants per cell. Co-Training improves τ2 task outcome over RL (Single) (0.70 vs 0.43), and both Verbalized Sampling and Co-Training improve P4G dialogue naturalness (4.33 and 4.45 vs 3.21 for RL Single).

The paper discusses mode collapse in LLMs (RLHF narrows LLM output distributions both empirically and structurally), LLM-based user simulation for RL (LLM simulators are the de facto training environment for dialogue and agentic RL), and multi-agent RL and co-training (self-play has driven gains in games and in LLM training; Liao et al. note its diversity ceiling, motivating dual-model co-training).

The authors identify simulator collapse as a structural failure for LLMs in multi-agent RL: LLM simulators are mode-collapsed, an RL policy trained against such a simulator inherits that narrowness, the policy's own entropy collapses onto the strategy that wins against the simulator's mode, and the resulting low-entropy policy fails to transfer to unseen simulators or real users.

They make three contributions. First, they formalize simulator collapse and show it is a structural failure of the training environment. Second, they give two complementary solutions: Verbalized Sampling at inference and Co-Training at training. They release SCOPE, an open framework that unifies multi-model rotation, self-play, and dual-model Co-Training behind one interface. Third, across Persuasion for Good, τ2-bench, and CooperBench, single-simulator RL's held-out success peaks early then drops back toward the untrained baseline by end of training; both solutions close most of the gap, and Population Co-Training takes the strongest held-out task success. All three settings are text-only, two-agent, English, and LLM-panel evaluated; whether the mechanism extends to N-agent populations, multimodal environments, or non-English settings remains open. Both solutions are simple because the bottleneck is in the environment rather than the algorithm.

Based on the paper, here are specific improvements I can implement in AI systems:

Improvement: Replace single frozen LLM simulators with a population of co-trained simulators.

Implementation:

• Maintain a FIFO buffer of 5 recent simulator checkpoints

• Sample a different simulator checkpoint per rollout batch

• Train both policy and simulator simultaneously on the same conversation data

• Use a curriculum reward for the simulator that targets 50% success rate to maximize variance

Resulting capability: The AI policy learns to handle diverse user behaviors instead of overfitting to one narrow response pattern. It generalizes better to unseen users and real-world interactions.

Multi-agent reinforcement learning for human-AI interaction typically relies on a single large language model to simulate user behavior. We show that this approach systematically fails to generalize, and trace the failure to simulator collapse: because the simulator LLM is mode-collapsed, an LLM policy trained against it overfits to narrow strategies that exploit the simulator's dominant mode, and such a policy transfers poorly to unseen simulators and real users. We formalize this collapse theoretically and propose two complementary solutions, one at inference time and one at training time. The inference-time solution, Verbalized Sampling, broadens the simulator's behavior by sampling from a verbalized response distribution, reducing mode collapse. The training-time solution, Co-Training, jointly optimizes the policy against a population of trainable simulators, preventing it from overfitting to any single simulator's mode. We validate both solutions on three multi-turn benchmarks: Persuasion for Good, tau squared-bench, and CooperBench. Verbalized Sampling improves held-out success by up to 9% over single-simulator RL, and Co-Training pushes gains further to 14%; the human study shows similar gain on real users. Both solutions preserve the policy diversity that collapses under single-simulator RL. To support further work in this direction, we release SCOPE, an open-source framework for Population Co-Training multi-agent RL. More broadly, our results suggest that the diversity of the training environment, not only the policy, is critical to the generalization of multi-turn RL to real-world deployment.