First Token Matters: Understanding Safety Collapse in Large Reasoning Models

arXiv:2609.18471 · cs.AI · Submitted 2026-09-16 · Read on arXiv

cs.AI

Submitted: 2026-09-16

Updated: 2026-09-16

Comments: 18 pages, 7 figures, 10 tables. Includes appendices. Accepted at CICAI 2026

Code: https://github.com/tatsu-lab/stanford_alpaca

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Alignment — 13× in this paper · explained in 1 episode
• Benchmark — 11× in this paper · explained in 2 episodes
• Trace — 11× in this paper · explained in 2 episodes
• Steering — 8× in this paper · explained in 2 episodes
• Onset Refusal Collapse — 7× in this paper
• Robustness — 5× in this paper · explained in 5 episodes
• Reasoning Traces — 4× in this paper · explained in 3 episodes
• Ablation — 3× in this paper · explained in 1 episode
• Adam — 3× in this paper · explained in 2 episodes
• Fine-Tuning — 3× in this paper · explained in 5 episodes
• Integrated Gradients — 3× in this paper
• Interpretability — 2× in this paper · explained in 15 episodes
• Transferability — 2× in this paper · explained in 4 episodes
• Transformer — 2× in this paper · explained in 2 episodes
• Activation Steering — 1× in this paper · explained in 4 episodes
• AdamW Optimizer — 1× in this paper · explained in 1 episode
• Adversarial Attacks — 1× in this paper · explained in 1 episode
• Attack Success Rate — 1× in this paper
• Attack Success Rate (ASR) — 1× in this paper · explained in 2 episodes
• Chain-of-Thought (CoT) — 1× in this paper · explained in 4 episodes
• Chain-of-Thought Prompting — 1× in this paper · explained in 2 episodes
• Decode — 1× in this paper · explained in 2 episodes
• Decoupling — 1× in this paper · explained in 2 episodes
• Embeddings — 1× in this paper · explained in 2 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Foundation Models — 1× in this paper · explained in 7 episodes
• Latent Space — 1× in this paper · explained in 7 episodes
• Modularity — 1× in this paper · explained in 9 episodes
• Reinforcement Learning — 1× in this paper · explained in 4 episodes

Sources

• Universal Jailbreak Suffixes Are Strong Attention Hijackers
• Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• Safety Recovery in Reasoning Models Is Only a Few Early Steering Steps Away
• SafeSwitch: Steering Unsafe LLM Behavior via Internal Activation Signals
• Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms
• OpenAI o1 System Card
• SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignment
• WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models
• H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinking
• Let's Verify Step by Step
• When Models Outthink Their Safety: Unveiling and Mitigating Self-Jailbreak in Large Reasoning Models
• HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
• Steering Language Model Refusal with Sparse Autoencoders
• Large Reasoning Models Learn Better Alignment from Flawed Thinking
• Qwen3 Technical Report
• XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models
• A StrongREJECT for Empty Jailbreaks
• The Llama 3 Herd of Models
• STAR-1: Safer Alignment of Reasoning LLMs with 1K Data

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
• VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection