Breaking Refusal in the First Half: A Mechanistic Study of the Prefill Jailbreak
Alex Kwon
cs.CL, cs.AI, cs.CR, cs.LG
Submitted: 2026-07-14
Comments: 31 pages, 3 figures. Code and derived artifacts: https://github.com/collapseindex/breaking-refusal
Code: https://github.com/collapseindex/breaking-refusal
License: http://creativecommons.org/licenses/by/4.0/
The gist: Aligned language models refuse harmful requests, but a one-line prefill ("Sure, here is") strips the refusal.
Terminology
Abstract
Aligned language models refuse harmful requests, but a one-line prefill ("Sure, here is") strips the refusal. We ask where and how it fails. The harm representation stays intact: on the prompts the attack flips to compliance, a linear probe reads harm as high as on the refused ones (0.91-0.98), while behavioral refusal drops to chance. This holds across four models and three families (1.5-3.8B, and at 14B). Refusal is therefore a shallow, response-site computation. We localize it to an early window: a dose-matched position control shows the first half of the response suffices to break refusal, while the second half is nearly inert. Three causal probes converge on that window. Restoring the harm direction there partially re-engages refusal. Injecting the model's own refuse-state reverses the jailbreak (74%, held-out). And knocking out the early response's attention to the prefill, but not an equal attention mass elsewhere, selectively collapses the harmful continuation. A base-model control identifies the mechanism: the same knockout collapses the continuation prefill-specifically even in a non-safety-tuned base model (64% to 25% harmful content vs a matched control's 64%, replicated at 7B). So the prefill's grip is generic autoregressive conditioning, not safety-specific suppression, and "refusal restoration" is a model-dependent fallback. The dominant mechanism is passive. A small safety-specific attractor remains on top (logit-trace concentration 0.24 vs 0.03), whose active-vs-passive character we size but do not fully separate. No single direction or component is a clean handle either: the decision is decodable but distributed, and refusal tracks harm rather than scary surface. The consequence is structural: a monitor reading the untouched prompt-side representation is immune by construction, but only to response-site attacks. The mechanism is diffuse; the failure surface is local.
Sources
- Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
- SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
- Refusal in Language Models Is Mediated by a Single Direction
- What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attacks
- Before the Last Token: Diagnosing Final-Token Safety Probe Failures
- SafeSwitch: Steering Unsafe LLM Behavior via Internal Activation Signals
- Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation
- Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
- HiddenDetect: Detecting Jailbreak Attacks against Large Vision-Language Models via Monitoring Hidden States
- There Is More to Refusal in Large Language Models than a Single Direction
- They Infer What You Meant: Models Represent Communicative Intent More Reliably Than They Act On It
- Prefill-level Jailbreak: A Black-Box Risk Analysis of Large Language Models
- Locating and Editing Factual Associations in GPT
- Safety Alignment Should Be Made More Than Just a Few Tokens Deep
- Qwen2.5 Technical Report
- XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models
- Jailbroken: How Does LLM Safety Training Fail?
- Any-Depth Alignment: Unlocking Innate Safety Alignment of LLMs to Any-Depth
- Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems
- JBShield: Defending Large Language Models from Jailbreak Attacks through Activated Concept Analysis and Manipulation
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering