On-Policy Distillation for LLM Safety: A Routing Approach to Template-Robust Realignment
Yongjian Guo, Wanlun Ma, Lingyu Shen, Xi Xiao, Sheng Wen
cs.AI, cs.CL, cs.CR, cs.LG
Submitted: 2026-07-29
License: http://creativecommons.org/licenses/by/4.0/
The gist: Fine-tuning is the dominant paradigm for specializing large language models (LLMs), yet it exposes a critical vulnerability: malicious data providers can embed harmful behaviors into downstream
Terminology
Abstract
Fine-tuning is the dominant paradigm for specializing large language models (LLMs), yet it exposes a critical vulnerability: malicious data providers can embed harmful behaviors into downstream corpora, creating models that retain professional skills while violating human values on demand. Existing safety-realignment defenses often fail in practice due to three key limitations: they frequently cause catastrophic forgetting of specialized skills; their effectiveness collapses when the defender cannot observe the attacker's prompt template; and successfully realigned models remain susceptible to re-jailbreaking via simple system prompt switches. To address these challenges, we propose Routing-based On-Policy Distillation (ROPD), a novel realignment framework that models the divergence between aligned and compromised output probability distributions rather than fitting specific prompt templates. We conduct extensive experiments comparing ROPD against four state-of-the-art baselines across three datasets and three base models with varying alignment strengths. Our results demonstrate that when baseline defenses face template mismatches, often accompanied by severe degradation in downstream task performance. In contrast, ROPD substantially mitigates template-mismatch risks, maintaining superior robustness in both defense effectiveness and capability preservation. While our analysis indicates ROPD is not entirely immune to template shifts, its performance degradation is negligible compared to existing methods, establishing a new standard for robust LLM realignment.
Sources
- Language Model Unalignment: Parametric Red-Teaming to Expose Hidden Harms and Biases
- Towards Healthy AI: Large Language Models Need Therapists Too
- Reinforcement Learning for LLM Post-Training: A Survey
- Of Models and Tin Men: A Behavioural Economics Study of Principal-Agent Problems in AI Alignment using Large-Language Models
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- TRACE: Distilling Where It Matters via Token-Routed Self On-Policy Alignment
- ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation
- Super-Exponential Approximation of the Riemann-Liouville Fractional Integral via Gegenbauer-Based Fractional Approximation Methods
- Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning
- A Survey of On-Policy Distillation for Large Language Models
- Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- Qwen2.5 Technical Report
- Gemma 2: Improving Open Language Models at a Practical Size
- Universal and Transferable Adversarial Attacks on Aligned Language Models
- Distilling the Knowledge in a Neural Network
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection