From Refusal Tokens to Refusal Control: Discovering and Steering Category-Specific Refusal Directions
cs.AI
Submitted: 2026-03-09
Updated: 2026-09-14
Comments: 10 pages, 23 with Appendix
Code: https://github.com/TransformerLensOrg/TransformerLens
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: Language models are commonly fine-tuned for safety alignment to refuse harmful prompts.
Terminology
Abstract
Language models are commonly fine-tuned for safety alignment to refuse harmful prompts. One approach fine-tunes them to generate categorical refusal tokens that distinguish different refusal types before responding. In this work, we leverage a version of Llama 3 8B fine-tuned with these categorical refusal tokens to enable inference-time control over fine-grained refusal behavior, improving both safety and reliability. We show that refusal token fine-tuning induces separable, category-aligned directions in the residual stream, which we extract and use to construct categorical steering vectors with a lightweight probe that determines whether to steer toward or away from refusal during inference. In addition, we introduce a learned low-rank combination that mixes these category directions in a whitened, orthonormal steering basis, resulting in a single controllable intervention under activation-space anisotropy, and show that this intervention is transferable across same-architecture model variants without additional training. Across benchmarks, both categorical steering vectors and the low-rank combination consistently reduce over-refusals on benign prompts while increasing refusal rates on harmful prompts, highlighting their utility for multi-category refusal control.
Sources
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Understanding intermediate layers using linear classifier probes
- Refusal in Language Models Is Mediated by a Single Direction
- OR-Bench: An Over-Refusal Benchmark for Large Language Models
- Constitutional AI: Harmlessness from AI Feedback
- Anisotropy Is Inherent to Self-Attention in Transformers
- The Llama 3 Herd of Models
- LEACE: Perfect linear concept erasure in closed form
- Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment
- PIQA: Reasoning about Physical Commonsense in Natural Language
- The Art of Saying No: Contextual Noncompliance in Language Models
- SCANS: Mitigating the Exaggerated Safety for LLMs via Safety-Conscious Activation Steering
- Measuring Massive Multitask Language Understanding
- WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models
- TruthfulQA: Measuring How Models Mimic Human Falsehoods
- Safety at Scale: A Comprehensive Survey of Large Model and Agent Safety
- The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets
- Steering Language Model Refusal with Sparse Autoencoders
- Training language models to follow instructions with human feedback
- Steering Llama 2 via Contrastive Activation Addition
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection