ExRole: From Team Trajectories to Executable Roles in Multi-Agent Language Models

ExRole: From Team Trajectories to Executable Roles in Multi-Agent Language Models

Abstract

Roles provide an interpretable interface for organizing language-model agents, yet most multi-agent systems treat them as hand-written prompt labels disconnected from learned behavior and parameter updates. We argue that a useful role should instead be an executable control variable: it should summarize behavior predictive of future utility, guide subsequent interaction, and identify the trainable capacity responsible for that behavior. We introduce ExRole, a trajectory-to-role framework that learns future-aware role prototypes from prefix-local team traces, resolves them into readable instructions and token-aligned role markers, and optionally routes shared LoRA rank slots with turn-aligned credit. Across MuSiQue and 2WikiMultiHopQA, ExRole improves over single-agent search by 15.0/14.4 and 13.5/16.1 EM/F1 points, respectively. Against the strongest non-ExRole controls, the corresponding gains remain 11.5/11.6 and 7.7/9.7 points. Across both benchmarks, the controlled results consistently favor trajectory-induced role conditioning over role-free, manual, random, and shuffled alternatives. Role-Agent-Turn interventions further show that the induced roles capture transferable behavioral specialization beyond fixed agent identities or turn positions.

Introduction

Large language model (LLM) agents increasingly solve knowledge-intensive tasks by interleaving reasoning with retrieval; recent multi-agent search work further distributes planning, retrieval, and synthesis across specialized agents. Roles offer a simple and interpretable abstraction for organizing this collaboration: they expose the intended division of labor without requiring a separate model for every agent. Role-playing designs commonly instantiate these functions as natural-language profiles fixed in prompts before execution, as in CAMEL's inception prompting. Figure 1 illustrates that changing such labels alone does not reliably improve answer quality or evidence use under a fixed policy. These labels are neither grounded in observed team behavior nor tied to the learning signal for role turns.

The paper asks: how can a system induce roles from prior collaboration and reuse them to coordinate new episodes and assign policy credit?

Learning roles from collaboration trajectories is more demanding than clustering surface-level action counts. First, a role representation must be inferred from information available at the current prefix while still capturing how that behavior affects future evidence and return; otherwise it either ignores delayed utility or leaks suffix information. Second, an induced cluster must become executable in a new episode: the same identity should guide the agent's natural-language instruction and remain recoverable inside the model. Third, shared team rewards create the familiar credit-assignment problem of identifying which action produced the outcome; here the credit must further reach the role turn and capacity path that produced a useful textual action.

The key insight is to treat a role as a shared control variable that connects behavioral abstraction with policy execution. This is instantiated in ExRole, an executable role-learning framework. ExRole first encodes prefix-local action, timing, interaction, and evidence statistics, and uses next-action, future-evidence, and final-return targets to organize trajectories by their prospective utility. It then summarizes each cluster as a reusable prototype and deterministically resolves the prototype into a readable instruction and a token-aligned role marker, without requiring an additional role-labeling language model. The resolved identity controls two complementary paths. At the interaction level, the instruction conditions the active agent and the marker remains aligned with the generated role-turn tokens. At the parameter level, ExRole-Routed combines that marker with the current semantic prefix to select a balanced sparse-delta gate over shared LoRA rank slots. GRPO is augmented with turn-aligned credit so that a useful role turn reinforces both its response tokens and, when routing is enabled, its selected capacity. ExRole-Shared retains the same induced roles and turn-aligned policy credit but uses a uniform shared LoRA path, providing a matched control that isolates the contribution of role-conditioned sparse LoRA routing.

Main Results

On MuSiQue, ExRole-Shared reaches 31.5 EM and 43.2 F1, improving single-agent search by 15.0 and 14.4 points and the strongest non-ExRole control by 11.5 and 11.6 points. On 2WikiMultiHopQA, ExRole-Routed reaches 50.0 EM and 59.7 F1, gains of 13.5 and 16.1 points over single-agent search and 7.7 and 9.7 points over the strongest control. Across both benchmarks, the controlled comparisons favor trajectory-induced roles over role-free, manual, random, and shuffled alternatives. Role-Agent-Turn interventions further show that the induced roles encode transferable action tendencies beyond fixed agent identities and turn positions.

Contributions

The paper's contributions are two-fold:

• It formulates LLM-agent role learning as trajectory-conditioned role induction, in which prefix-local behavior and future-utility targets define reusable role prototypes.

• It makes induced roles executable by binding each prototype to a readable instruction, a token-aligned role marker, and a deterministic agent assignment, and connects that identity to role-conditioned sparse LoRA routing with turn-aligned credit.

Method Overview

ExRole addresses the gap between discovering recurring team behaviors and turning them into specialization that can be executed and optimized in later episodes. Given prior collaboration trajectories, the goal is to induce roles that capture future-relevant behavior, bind each role to a consistent interaction and model-side identity, and assign learning credit to the role and capacity path that produced an action. The pipeline consists of four components: predictive trajectory encoding and clustering produce a role library; deterministic role binding makes each prototype executable; role-conditioned interaction and sparse LoRA routing apply the same identity during generation; and turn-aligned grouped reinforcement learning updates the policy and, when enabled, the router.

Future-Aware Role Induction

A useful role should summarize not only what an agent has done, but also what its behavior predicts about subsequent collaboration. For each logged agent turn, ExRole constructs a prefix-local behavior vector which summarizes action tendencies, turn position, team context, evidence use, and answer behavior observed up to that point. A role encoder produces an embedding and predicts the same agent's next action, future evidence utility, and final trajectory return using cross-entropy, binary cross-entropy, and squared-error terms. This predictive objective shapes the embedding with future utility without placing suffix information in its input.

For each candidate role count, Euclidean K-means is run from each configured seed and every sufficiently supported cluster is summarized as a source prototype. When K is not fixed by the execution budget, it is selected by a separation–stability criterion. Finally, a deterministic joint resolver scores all surviving prototype–template pairs, assigns distinct functional types when possible, and converts the assignments into readable instructions and token-aligned role markers. Role names are interpretations of learned behavioral clusters rather than manually specified training labels.

Executable Role Binding

Induced clusters become useful only when they control subsequent team behavior. ExRole selects at most N source prototypes, remaps them to contiguous runtime role IDs, and pads missing entries with a generic collaborative role. The resolved role conditions the observation presented to the active agent. The marker remains in the token sequence and defines role-turn segments. This construction makes the role used in the prompt identical to the role consumed by the model-side router.

Role-Conditioned Sparse Routing

Prompt conditioning alone does not determine which trainable capacity supports a role. ExRole-Routed therefore modulates the shared LoRA update at the rank-slot level. The router combines the executable-role feature, its discrete identity, and a pooled semantic prefix. The scores define a probability distribution and a hard budget of directly selected slots. The hard mask concentrates role-specific deviations, while centering preserves a balanced shared update before clipping. Both variants retain the same LoRA parameters and role-conditioned interaction; they differ only in the model-side routing state.

Turn-Aligned Optimization

An episode-level reward alone assigns the same credit to every response in a team trajectory. ExRole retains the group-relative trajectory advantage from GRPO, while adding discounted local credit to the response tokens produced at each turn. The router learns to predict this target and associate positive role-turn credit with its selected capacity. Load-balance, role-diversity, sparsity, and entropy terms regularize the remaining routing distribution, while Kullback–Leibler regularization anchors the policy. The two model variants are optimized under the same turn-aligned policy objective and differ only by the router loss.

Experimental Setup

The evaluation uses MuSiQue and 2WikiMultiHopQA, which require multi-step evidence composition over supporting-document collections. The controlled comparison includes single-agent search, multi-agent systems without roles or with hand-written roles, and random or shuffled role controls. All controlled systems share the same policy backbone, action interface, and evaluation protocol. ExRole-Shared and ExRole-Routed additionally share the induced role library and training objective, differing only in whether LoRA capacity is uniformly shared or role routed. Metrics include Exact Match, token-level F1, and strict success.

Role Mechanism

The induced Researcher, Analyst, and Verifier roles differ in search, evidence, answer, and timing statistics, and their trajectory embeddings form separated regions. Their scheduled turns also produce distinct action and routing profiles.

Role-Agent-Turn Disentanglement

Using fixed checkpoints, the paper orthogonally varies speaker order and agent-role assignment on MuSiQue. Role identity reduces held-out action log loss by 19.1% for Shared and 18.7% for Routed after accounting for agent identity and turn phase. Between-role Jensen–Shannon distance is 48–60 times the within-role cross-agent distance, while Researcher, Analyst, and Verifier search rates are 96.6%, 85.8%, and 61.3%. These results indicate transferable action tendencies rather than fixed agent or turn labels.

External Reference

ExRole-Shared reaches 49.0/59.1 EM/F1 on 2WikiMultiHopQA and 31.5/43.2 on MuSiQue; ExRole-Routed reaches 50.0/59.7 and 30.0/41.5. ExRole is competitive with several MAS-RL systems but remains below the strongest search-specialized policies.

Capacity-Path Analysis

Both capacity paths preserve the induced-role gains over the corresponding single-agent and no-role controls, while their relative ordering varies across benchmarks. The auxiliary and future-utility losses remain active throughout training, while turn credit varies with sampled trajectories. Both variants receive turn-level credit throughout optimization; only Routed additionally receives the router losses.

Conclusion

ExRole turns team trajectories into executable roles for interaction and turn-aligned optimization. On MuSiQue and 2WikiMultiHopQA, induced roles yield distinct search, synthesis, and verification behaviors and outperform single-agent and role-free controls. Role-Agent-Turn interventions show that these behaviors generalize beyond fixed agents and turn positions, supporting trajectory-induced role binding as a practical framework for structured language-agent collaboration. This interface makes learned specialization explicit and reusable during policy optimization.

Improvements to AI Systems:

1. Executable Role Induction from Trajectories – Replace hand-written prompt roles with roles learned from past team behavior. The system can automatically discover reusable behavioral specializations (e.g., Researcher, Analyst, Verifier) from prefix-local action/evidence statistics, without manual labeling.

2. Future-Aware Role Prototyping – Train role embeddings to predict next actions, future evidence utility, and final returns. This enables the AI to summarize not just what an agent did, but what its behavior predicts about downstream success, improving coordination in new episodes.

3. Deterministic Role-to-Instruction Binding – Convert learned clusters into readable natural-language instructions and token-aligned role markers without an extra LLM. This allows the same role identity to condition both the prompt and the model’s internal routing, making specialization interpretable and reproducible.

4. Role-Conditioned Sparse LoRA Routing – Dynamically select shared LoRA rank slots based on the active role and semantic context. This lets a single model allocate distinct trainable capacity per role, improving parameter efficiency and specialization without adding separate models.

5. Turn-Aligned Credit Assignment – Replace episode-level rewards with per-turn discounted credit, so the system can attribute outcomes to specific role turns and capacity paths. This improves reinforcement learning stability and enables fine-grained policy updates for each role.

6. Disentangled Role-Agent-Turn Generalization – Use induced roles as transferable control variables independent of fixed agent identities or turn positions. The system can re-assign roles across agents and reorder turns without losing behavioral specialization, enabling flexible team composition.

7. Balanced Sparse-Delta Gate with Regularization – Apply load-balance, role-diversity, sparsity, and entropy penalties to the routing distribution. This prevents capacity collapse, ensures all roles receive training signal, and maintains a balanced shared update across rank slots.

8. Controlled Role Evaluation Protocol – Systematically compare trajectory-induced roles against role-free, manual, random, and shuffled baselines under identical policy backbones. This provides a rigorous method to validate that learned roles—not prompt labels or agent identities—drive performance gains.

What the Improved AI System Can Do:

• Automatically discover and reuse specialized team behaviors from past collaboration logs, improving multi-step reasoning and retrieval tasks (e.g., +11.5 EM over strongest non-role controls on MuSiQue).

• Generate interpretable role instructions and token markers that condition both the agent’s prompt and its trainable LoRA capacity, enabling transparent division of labor.

• Optimize policy and router jointly with turn-level credit, allowing precise reinforcement of actions that lead to successful evidence composition and answer synthesis.

• Generalize role behaviors across different agents and turn orders, making the system robust to team reconfiguration.

• Achieve competitive performance on multi-hop QA benchmarks (up to 50.0 EM / 59.7 F1 on 2WikiMultiHopQA) while maintaining a single shared model backbone with sparse parameter routing.