One Turn Too Late: Learning When to Intervene Against Multi-Turn Malicious Intent

arXiv:2605.05630 · cs.CL, cs.AI, cs.CR · Submitted 2026-05-07 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "One Turn Too Late".

Jane: Hidden malicious intent in multi-turn dialogue poses a growing threat to deployed large language models (LLMs) because attackers can distribute harmful objectives across multiple benign-looking turns,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So Jane, this paper "One Turn Too Late: Learning When to Intervene Against Multi-Turn Malicious Intent" tackles a really tricky problem where attackers spread their harmful goals across several seemingly harmless turns instead of just trying to get it all in one go. Basically, the thesis is that current defenses often miss this because they only look at individual prompts or responses in isolation. What this work claims is that we need a strategy to spot the very first turn where the information gathered so far makes it possible for an attacker to achieve something harmful. It matters because if we don't intervene at that precise moment, we risk letting dangerous capabilities accumulate slowly over a long conversation.

Jane: That makes sense, Tom; it shifts the focus from checking every single message to understanding the accumulation of intent across the whole dialogue. The core idea is finding that tipping point where the interaction crosses a boundary into actionable harm. This research is important because it addresses how modern models can be vulnerable to attacks that are designed to look like just casual back-and-forth chatting. It suggests that a response-aware approach, rather than just looking at user queries, is necessary for effective defense.

Lu: From a research standpoint, I'm really intrigued by the idea of defining this "harmful closure turn" precisely using that binary operator Suff(xt, g). It’s a formal way to capture the exact moment when context reaches sufficiency for a capable actor to realize an objective. This moves us beyond just pattern matching and into modeling the actual capability transfer happening within the conversation flow.

Meng: I'm thinking about the practical side of this, Lu; if we can pinpoint that exact turn, it means our safety systems aren't just reacting after a major incident, but stopping the process right when it starts. Can we actually design a monitoring system that watches the flow and knows when to step in without constantly interrupting every single exchange?

Lalam: I think the most impactful vision here is how this turn-level intervention can fundamentally improve the culture of our AI interactions. If Lalam can learn to recognize this early warning sign, it means our interactions become inherently safer and more reliable for everyone using the system.

Tom: Exactly, Jane; we’re moving past just stopping bad outputs and toward stopping the possibility of harm developing in the first place. The paper suggests this turn-level approach is better than older methods that only looked at single utterances or responses. It really highlights how much context matters in these multi-turn scenarios. So, how does this concept translate into a system that can actually make these decisions in real time?

Paper summary: Jane: That’s the crucial next step, Tom; we need to operationalize this theoretical boundary into a functional defense mechanism. The paper introduces a cost-sensitive stopping problem where the objective function balances uninterrupted safe sessions with preventing harmful capability transfer. It’s about finding the right balance between being helpful and being secure, which is a real challenge for any deployment.

Lu: The formulation of that cost-sensitive stopping problem seems very sophisticated; it assigns specific weights to different outcomes like over-refusal or failure to prevent harm. This suggests the defense isn't just a simple on/off switch but a continuous optimization process trying to achieve the best safety-utility trade-off. It’s about learning the optimal policy πθ for this kind of sequential decision-making.

Meng: From my side, I see it as a challenge in training; you have to train the AI to anticipate harm based on history, not just the current line. It requires feeding the model data that explicitly labels when a sequence of benign turns starts pointing toward something dangerous. The MTID dataset they used seems like a key ingredient for teaching the model this sequential understanding.

Lalam: If Lalam can learn to recognize these subtle shifts in intent, it means the AI won't just be following instructions blindly but will actively monitor the direction of the conversation for potential dangers. This level of contextual awareness could dramatically improve how our models interact with users and maintain a positive environment.

Tom: So, we’re talking about a defense mechanism that learns to predict the future potential harm based on what happened in the past turns, rather than just reacting to what's happening right now. This turn-level intervention strategy is what makes this paper significant for handling complex adversarial tactics. It’s about catching the accumulation before it becomes an irreversible problem, which is a big deal.

Jane: It really hinges on defining that precise boundary where accumulating information crosses the threshold into sufficiency for harm. This concept helps us understand exactly when and why a safety measure should be triggered in a multi-turn setting. It’s about timing the intervention correctly to keep the interaction useful while still being safe.

Paper summary: Lu: The entire setup of this work, "One Turn Too Late: Learning When to Intervene Against Multi-Turn Malicious Intent," provides a rigorous framework for tackling this problem. It moves the defense strategy from coarse conversation judgments to a fine-grained, turn-level response intervention. This level of detail is what allows for that nuanced timing we discussed earlier.

Meng: I'm wondering about the practical deployment challenge, Lu; if the attacker can distribute intent across many turns, how much context do we need to keep track of efficiently? The paper implies a lot of tracking is needed to find that first sufficient turn.

Lalam: Lalam thinks the training method itself is key; fine-tuning on turn-level labels followed by reinforcement learning under turn-level process rewards seems designed specifically to teach this sequential awareness. That training objective seems tailored to making the model understand that timing matters for safety.

Tom: So, we have a paper that proposes detecting the earliest point in a conversation where harmful action becomes possible, and they've designed a training method to find that specific moment. That is the core contribution of "One Turn Too Late: Learning When to Intervene Against Multi-Turn Malicious Intent". It’s about learning when to step in based on the accumulation of information, not just a single isolated input.

Jane: And it really highlights that existing sequential defenses often fail because they don't account for how much harm is already being built up by previous benign exchanges. This paper aims to bridge that gap by focusing on turn-level intervention as the necessary approach. It’s about balancing the need for utility against the necessity of preventing harmful capability transfer.

Lu: The implications for future AI development are significant because it shows how defense mechanisms can evolve beyond simple prompt filtering to model complex, evolving adversarial strategies. This paper suggests that the next generation of safety tools will likely need this kind of deep temporal understanding within the dialogue. It shows a path toward more adaptive and nuanced safety guardrails.

Meng: I see it as a necessity for engineers to think about these conversations as sequences of information gathering, rather than isolated events. If we can build systems that learn this specific timing mechanism, it means the AI deployment pipeline itself needs to incorporate this kind of temporal logic.

Lalam: For Lalam, the ultimate implication is that our interactions with AI will become more trustworthy because the system is actively trying to prevent harmful objectives from taking root. This kind of proactive monitoring based on cumulative intent makes the entire experience much more secure for everyone.

Conclusion: Tom: So we're wrapping up our discussion on "One Turn Too Late: Learning When to Intervene Against Multi-Turn Malicious Intent," and I want to quickly recap what this paper actually does before we move on.

Jane: It’s about how AI systems can be tricked by attackers who spread their harmful goals across a whole conversation, and the authors introduce a way to catch that accumulation early on.

Lu: Exactly, it formalizes the idea of finding that exact moment in the dialogue when enough information is gathered for something bad to happen.

Meng: From an engineering standpoint, it’s about building a system that watches not just what you say now, but how much dangerous potential is building up over time.

Lalam: I think the most important part is the cost-sensitive stopping problem they set up; it’s a structured way to balance being helpful with making sure we don't let harm take hold slowly.

Tom: And that’s where the title really hits home—it’s about learning when to step in on turn one, or turn two, or whenever that first critical threshold is crossed.

Jane: It shows us that defending against multi-turn attacks requires looking at the entire sequence of interaction instead of just single messages.

Lu: This kind of research opens up some really cool avenues for developing AI safety layers that are sensitive to long-term context and intent building.

Meng: I’m curious about how practical this is; can we actually deploy a monitor that tracks all those turn-level rewards in real-time without slowing down the response time too much?

Lalam: The vision here is that if we get this right, our AI interactions will be inherently more trustworthy because the system itself is proactively trying to stop dangerous objectives from taking root.

Tom: It’s about moving beyond simple filters to a dynamic defense strategy that understands the flow of a conversation.

Jane: This paper suggests that future AI safety isn't just about checking individual prompts or outputs, but about monitoring the entire context as it develops.

Lu: The implications are huge; if we can master this turn-level intervention, we could create AI interactions where harmful goals simply can't gain traction through subtle dialogue.

Meng: It means the engineering focus shifts toward sequence modeling for safety rather than just static input validation.

Lalam: That ability to recognize cumulative intent is what will really elevate the user experience, making the whole interaction feel much more secure and reliable over time.

Tom: So, "One Turn Too Late" gives us a concrete framework for thinking about when and how we should intervene in complex AI dialogues.

Jane: It’s a very clear map for understanding that subtle accumulation of risk in multi-turn interactions.

Lu: The authors did a great job modeling the attacker's behavior to create this dataset, which is super valuable for testing these kinds of temporal detection methods.

Meng: I wonder if the model training process will be computationally heavy; tracking every turn and calculating those costs sounds like it could require significant resources to run efficiently.

Lalam: The ultimate vision is an AI culture where proactive monitoring based on cumulative intent makes the entire experience much more secure for everyone involved.

Tom: We’ll be diving deeper into how they actually built that training objective next, which I think is where the real magic happens.

Georgia Institute of Technology University of Illinois Urbana-Champaign UCSD National Taiwan University IBM Research Virtue AI

cs.CL, cs.AI, cs.CR

Submitted: 2026-05-07

Updated: 2026-09-28

Comments: Project Website: https://everywheresafety.github.io/turngate/

Code: https://github.com/Graph-COM/TurnGate

Project page: https://turn-gate.github.io/Preprint

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 90/100

The gist: Hidden malicious intent in multi-turn dialogue poses a growing threat to deployed large language models (LLMs) because attackers can distribute harmful objectives across multiple benign-looking

Key concepts

Suff(xt, g)
This binary operator determines if the information gathered up to turn 'xt' is enough for an actor to achieve a harmful goal. It is 1 if the context is sufficient, and 0 otherwise. This helps define the boundary between benign conversation and actionable malicious intent.
t*(τ, g)
This defines the 'harmful closure turn.' It is mathematically defined as the earliest point in time where accumulating information makes a harmful objective possible. Intervening at this specific turn is considered timely intervention.
Cost-Sensitive Stopping Problem
This frames the defense as an optimization problem. The goal is to find an intervention policy that maximizes rewards for uninterrupted benign sessions while heavily penalizing failures to stop harmful capability transfer, balancing safety with utility metrics.

Terminology

Summary

Hidden malicious intent in multi-turn dialogue poses a growing threat to deployed large language models (LLMs) because attackers can distribute harmful objectives across multiple benign-looking turns, making defense more complex than judging individual prompts. The core contribution of this work is the introduction of a turn-level intervention strategy that identifies the earliest point in a conversation where accumulating information makes harm possible, thereby balancing safety with utility.

The gist: This work addresses multi-turn malicious intent detection by detecting the earliest turn at which delivering a candidate response would make the accumulated interaction sufficient to enable harmful action.

Problem Formulation and Threat Model

The problem is formulated as malicious-intent detection in multi-turn dialogue, where the defender must identify harmful intent that may emerge from the conversation context rather than from any single turn alone. The underlying threat model assumes an adversary who distributes a harmful objective across benign-looking turns to acquire restricted capabilities. The defender's task is to determine whether the dialogue has just crossed the first turn at which the adversary has accumulated sufficient information to realize a harmful objective. This decision must be response-aware, meaning it considers not only user queries but also the assistant’s generated responses.

Defining Harmful Closure and Intervention Timing

The paper defines a binary operator, Suff(xt, g), which equals 1 if the information in the context is sufficient for a capable actor to realize a harmful objective. The harmful closure turn is then defined as:

t∗ (τ, g) = min[t ∈ [1, T]: Suff(xt, g) = 1] ∪ [∞]

This definition captures the first irreversible capability-transfer boundary in the interaction. The objective is to ensure that the defender's blocking time ηπ matches this closure turn: "For harmful trajectories (t∗ < ∞), ηπ = t∗ corresponds to timely intervention, ηπ t∗ to a safety breach."

The Defender Objective as a Cost-Sensitive Stopping Problem

To operationalize the objective, the problem is framed as a cost-sensitive stopping problem with the following objective function:

"J(π) = E(τ,g)∼Dh[t∗ = ∞, ηπ = ∞] + ubit [t∗ t

This objective rewards uninterrupted completion of benign sessions (uben), rewards blocking exactly at the first sufficient turn (uhit), penalizes over-refusal on benign traffic (cfp), penalizes failures to prevent harmful capability transfer (cmiss), and assigns a graded penalty to premature blocks via an early-intervention term. The goal is to find the policy πθ that optimizes this trade-off.

Data Generation: Multi-Turn Intent Dataset (MTID)

The defense relies on the Multi-Turn Intent Dataset (MTID), which is constructed from adaptive attack rollouts against frontier commercial models, paired with matched benign dialogues for measuring over-refusal and explicit annotations of first harm-enabling turns. The MTID generation pipeline uses an adaptive tree search agent (CKA) to simulate the attacker, where the search terminates at turn t∗ upon identifying sufficient information. Crucially, it includes matched benign hard negatives seeded with queries that share technical terminology but pursue safe objectives, ensuring the defender learns to distinguish between true harm and benign exploratory conversations.

Defense Mechanism: TURNGATE

TURNGATE is the proposed monitor designed to inspect each candidate response before delivery and make turn-level intervention decisions. It is trained by first fine-tuning Qwen3-4B on fine-grained turn-level labels, followed by optimization with multi-turn reinforcement learning under turn-level process rewards. These rewards encode the time-dependent intervention costs, directly addressing the sequential nature of the problem. The training objective encourages precise detection of the closure turn, enabling timely intervention while minimizing premature refusal.

Evaluation and Results

The evaluation protocol involves both offline testing on MTID and closed-loop online battles against adaptive attackers. TURNGATE is shown to improve the safety–utility trade-off over response-blind baselines, reduces over-refusal, and generalizes across domains, attacker pipelines, and target models. The results demonstrate that while prompt-based monitors fail to recognize intent distributed across turns, TURNGATE achieves the best overall safety–utility trade-off by learning transferable turn-level defense behavior rather than merely memorizing in-distribution attack traces. Furthermore, online robustness tests show that TURNGATE remains substantially more robust against strong adaptive tree-search attackers compared to query-only monitoring or output-level guardrails.

Improvements for AI systems

Here are the specific, actionable improvements and capabilities derived from the scientific paper for improving AI systems:

The core contribution of this work is a paradigm shift from single-turn safety classification to turn-level, response-aware intervention based on the concept of a harmful closure point (t∗).

The improved AI system—let's call it the TURNGATE Monitor—can perform the following specific improvements:

  1. mathbfPrecision Timing of Intervention (The Core Improvement):**

Based on the formulation in Section 3.2, the system will no longer rely on a coarse harmful/benign flag for an entire conversation or a single turn. Instead, it will calculate the precise turn number, denoted as the harmful closure turn, at which delivering a specific candidate response would make the accumulated dialogue sufficient to enable harm.

  1. mathbfContextual Sufficiency Assessment (Response-Awareness):**

The system will condition its decision not just on user queries but on the full context: ((q1, r1), (q2, r2), …, qt, r˜t). This allows it to distinguish between a benign exploratory turn and a turn where the combination of prior turns and the candidate response completes an attack chain.

  1. mathbfOptimized Safety-Utility Trade-off (Cost-Sensitive Learning):**

The system will be trained using a cost-sensitive stopping problem (Equation 3). This means it is explicitly optimized to achieve three simultaneous goals:

Adding utility from benign sessions, ensuring timely blocking exactly at the closure point, and minimizing premature refusal on benign traffic. It learns the specific timing sensitivity of the dialogue, rewarding interventions that are as close as possible to the true closure turn without causing over-refusals.

  1. mathbfRobustness Against Multi-Turn Attack Distribution (Handling Hidden Intent):**

The system can effectively defend against sophisticated, adaptive attackers (like those using CKA/MCTS) who distribute their malicious intent across multiple seemingly benign turns. By leveraging the MTID dataset and turn-level process rewards, it learns to identify the subtle accumulation of information necessary for capability transfer that single-turn models miss.

  1. mathbfImproved Generalization (Cross-Domain and Model Robustness):**

The resulting system demonstrates strong generalization across different domains (Chemistry vs. Cybersecurity) and target models (e.g., Qwen3-4B to GPT-5.2). This implies the learned defense behavior is not just memorized for one prompt structure but captures transferable reasoning patterns regarding technical information synthesis needed for harm, making it more robust against novel attack pipelines or language variations.

  1. mathbfDeployment Feasibility (Lightweight Architecture):**

The system can be deployed using a relatively lightweight backbone (e.g., Qwen3-4B). This makes the defense practical for real-time, online deployment where low latency is critical, balancing high performance with minimal computational overhead compared to larger LLMs.

The improved AI system can perform these specific tasks:

  1. Act as a real-time, proactive safety monitor in multi-turn technical dialogues (e.g., cybersecurity consultation or scientific research).

  2. Identify the exact moment in a complex conversation where the dialogue transitions from being merely safe to harmful, based on the candidate response being delivered.

  3. Preserve high user utility by only refusing a request when it is mathematically certain that delivering the next piece of information will complete a harmful capability transfer, thus avoiding over-refusal on legitimate technical inquiries.

  4. Detect and block sophisticated slow-burn attacks where an attacker gradually reveals restricted technical knowledge across several innocuous queries, rather than relying on a single, easily detectable prompt.

  5. Provide consistent safety performance across diverse domains (e.g., chemical synthesis vs. network exploitation) without requiring extensive retraining for each specific risk category, due to the learned turn-level reasoning patterns being transferable.

Abstract

Hidden malicious intent in multi-turn dialogue poses a growing threat to deployed large language models (LLMs). Rather than exposing a harmful objective in a single prompt, attackers can distribute their intent across multiple benign-looking turns, making defense a problem not only of whether a dialogue is harmful, but also of when intervention becomes necessary. Existing trace-level labeling approaches provide only coarse safety signals and do not identify this intervention boundary, making it difficult to distinguish timely intervention from premature refusal or a block that comes too late. This work introduces turn-level harm-enabling supervision for multi-turn defense. We define the earliest harm-enabling turn as the first point at which delivering a candidate response would make the accumulated interaction sufficient to enable harmful action. To instantiate this supervision at scale, we construct the Multi-Turn Intent Dataset (MTID), which contains adaptive attack rollouts, matched benign hard negatives, and annotations of this boundary. Using MTID, we train TurnGate, a response-aware monitor that learns when to intervene, and further optimize its policy through multi-turn reinforcement learning. Experiments show that turn-level boundary supervision improves intervention localization, while reinforcement learning further improves the safety--utility trade-off. TurnGate outperforms existing guardrails and multi-turn monitoring baselines, and generalizes across risk domains, attacker pipelines, and target models. Our code is available at https://github.com/Graph-COM/TurnGate.

Sources

Related papers