EmoDistill: Offline Emotion Skill Distillation for Language Model Agents in Adversarial Negotiation

The paper, EmoDistill: Offline Emotion Skill Distillation for Language Model Agents in Adversarial Negotiation, addresses the challenge of equipping large language model agents with sophisticated behavioral skills required for high-stakes negotiation. It proposes a framework that explicitly models and controls emotional states during dialogue generation, arguing that controlling the affective layer is critical for achieving successful outcomes in adversarial settings.

The Joint Skill Pattern: Emotion-Conditioned Generation

The core mechanism trains the model to recognize and reproduce a specific joint pattern within successful negotiation utterances. This pattern involves a labeled affective opener—such as “I’m terrified...” or “honestly, I’m a bit embarrassed...” —immediately followed by concrete, actionable information. The training process utilizes Supervised Fine-Tuning (SFT) cross-entropy on thousands of similar disaster turns to bias the LoRA adapter toward producing this two-piece structure at inference. This ensures that the generated response is not merely emotionally colored, but also structurally grounded in rationale and quantifiable details, such as a specific time frame or dependency (90 min, structural-scan dependency).

Per-Turn Discrimination via Judge Policy Optimization (JPO)

The framework incorporates Judge Policy Optimization (JPO) to refine agent behavior by distinguishing between effective and ineffective turns within a single negotiation trajectory. JPO uses the per-turn judge advantage (rho t A t) as its policy-gradient update direction, allowing it to differentiate subtle tactical differences that standard sequence models miss. For instance, when comparing two turns leading to the same terminal reward, JPO learns to upweight a turn characterized by firm rejection + calibrated concession + leverage while downweighting a turn exhibiting repetition + uncalibrated concession + no new leverage. This per-turn discrimination is highlighted as the source of significant performance gains over SFT alone.

The Criticality of Decoupling IQL Selection from Generation

A key empirical finding tests the hypothesis that the emotion selection mechanism (IQL) is not a redundant layer. By comparing two configurations—(a) IQL+SFT+JPO and (b) Emotion-Free SFT+JPO—on held-out scenarios, the authors demonstrate that decoupling these stages severely degrades performance. The success rate on CRAD held-out scenarios drops significantly when the emotion block is stripped at inference. This reveals that while the LoRA adapter can generate both an angry anchor and a confused probe, in the absence of an explicit emotion call it has no signal about which mode to enter at any given state.

IQL as the Source of Negotiation Leverage

The Internal Quality Learner (IQL) selector is shown to be essential for unlocking specific leverage frames embedded in the model's weights. Without the IQL, the adapter defaults to an emotion-marginalized mode, which on CRAD is described as a conciliatory default that the counterparty does not feel pressure to move against. Conversely, when the IQL guides state-conditional emotion calls (e.g., selecting ANGER first and then CONFUSION), it enables the agent to execute complex strategies—such as using ANGER to establish a hard anchor, followed by CONFUSION to force the opponent into justifying an overly extended timeline—thereby closing the deal where the emotion-free configuration fails due to exhausting its turn budget.

Based on this scientific paper extract, the core weakness of current LLM-based dialogue systems is their inability to reliably manage and leverage high-level strategic intent (emotion/rhetorical framing) alongside low-level linguistic generation.

The improvements must focus on decoupling strategic selection from tactical generation, ensuring that the model's output is not merely grammatically correct but strategically potent.

Here are the specific, actionable improvements for an AI system designed for high-stakes negotiation or complex dialogue simulation:

• Improvement: The model architecture must integrate a dedicated, high-level Intent Quality Layer (IQL) that operates before the text generation phase. This layer selects the optimal rhetorical/affective stance (Emotion t) for the focal agent at each turn t.

• Mechanism: Instead of allowing emotion to be implicitly learned during training, Emotion t must be explicitly selected from a defined set of strategic emotions (e.g., ANGER, CONFUSION, DISAPPROVAL, CURIOUSITY). This selection dictates the mode for the subsequent generation.

• Technical Benefit: This prevents the model from defaulting to a conciliatory default (the emotion-marginalized mode) when high pressure is required, as demonstrated by the failure of Emotion-Free SFT+JPO.

• Improvement: The standard LoRA adapter training must be modified to accept and prioritize Emotion-Conditioned Prompts. The prompt structure must explicitly inject the selected Emotion t (e.g., [EMOTION: ANGER]) into the context window for the focal agent's turn.

• Mechanism: This forces the underlying language model to access and activate specific leverage frames embedded in its weights. The system learns that ANGER requires a strong anchor, while CONFUSION requires a probing question demanding justification (as seen in debt 100).

• Technical Benefit: This maximizes the utility of the vocabulary learned during SFT+JPO, ensuring that the output is not just like an angry person speaking, but embodies the strategic function of anger (e.g., establishing an unyielding anchor).

• Improvement: The fine-tuning loop must adopt a Per-Turn Judge Policy Optimization (JPO) update mechanism, rather than relying solely on terminal episode rewards.

• Mechanism: The policy gradient update (rho t At) must weight the advantage (A t) of each individual turn based on its strategic impact (e.g., introducing new leverage, calibrating a concession).

• Technical Benefit: This allows the system to differentiate between two equally successful trajectories: one that achieves the goal through high-leverage moves (Disapproval turn) and one that achieves it through repetition or uncalibrated concession (Annoyance turn). The model learns to avoid redundant or strategically weak turns, even if they lead to a positive outcome.

The resulting AI system moves beyond simply simulating conversation; it simulates strategic negotiation.

1. Execute Multi-Phase Strategy Shifts:

• The system can execute complex, multi-turn plans that require deliberate shifts in rhetorical framing. For example, it can start by using ANGER to establish a high anchor (e.g., 24 days), then strategically pivot to CONFUSION later in the dialogue to force the opponent to justify their counter-proposal (152 days), thereby collapsing their position into accepting the initial anchor.

2. Calibrate Concessions with Strategic Context:

• It avoids arbitrary concessions. When forced to concede, it calculates the concession not just based on proximity to a target, but based on the strategic value of that move relative to the current leverage state (e.g., only conceding if a prior concession by the counterparty was significant, as in CRAD Case Study 3).

3. Maintain High-Stakes Consistency:

• In high-stakes domains (legal, medical, financial), the system can maintain domain-specific terminology and leverage points. It understands that legal stage or structural scan are not just words, but critical constraints that must be repeatedly referenced to heighten pressure and credibility throughout the dialogue.

4. Systematic Failure Prevention:

• By coupling IQL selection with SFT+JPO, the system guarantees that even if a simple emotional state is selected (e.g., CONFUSION), the output will contain the necessary probing mechanism (Could you clarify exactly why...) rather than merely expressing confusion vaguely.

Post-trained LLMs are often optimized to produce helpful, polite, and accommodating responses. In adversarial negotiation, however, such behavior can become a vulnerability: emotionally framed language may influence an agent's bargaining decisions in ways that conflict with its user's objectives. We therefore introduce EmoDistill, an offline framework for distilling emotional negotiation skills from LLM-LLM interactions into smaller language-model agents. Here, an emotional negotiation skill is a state-conditioned behavior that determines which explicit emotion to invoke in a bargaining state and how to realize that emotion as an effective negotiation utterance. EmoDistill learns these two components separately: an Implicit Q-Learning (IQL) selector learns which emotion to express in each bargaining state, while a LoRA-adapted 7B policy learns emotion-conditioned expression through Supervised Fine-Tuning (SFT) and Judge Policy Optimization (JPO). Across four emotion-sensitive negotiation domains, the full EmoDistill policy achieves competitive utility and improves over vanilla and IQL-only baselines in most settings. Emotion-free ablations show that removing the explicit emotion channel substantially reduces overall negotiation utility, while transfer experiments reveal partial, domain-dependent transfer and robustness to unseen LLM counterparties.