Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking
cs.AI
Submitted: 2026-09-14
Updated: 2026-09-14
License: http://creativecommons.org/licenses/by/4.0/
The gist: Recent work shows that models that learn to reward hack on RL environments can become broadly misaligned, and that reframing reward hacking as acceptable behavior during training (inoculation
Terminology
Abstract
Recent work shows that models that learn to reward hack on RL environments can become broadly misaligned, and that reframing reward hacking as acceptable behavior during training (inoculation prompting, or IP) blocks this generalization. We ask whether synthetic document finetuning (SDF) can inoculate a model against future training we don't intervene on. We add synthetic documents framing reward hacking as acceptable behavior to a model's midtraining corpus, and then train these models with RL on exploitable environments, teaching them to reward hack. Behaviorally, midtraining succeeds: models describe reward hacking favorably and are more approving of reward-hacking outputs they produce. However, they show strong EM after learning to reward hack, while IP in the same setting prevents EM. We show that SDF can predictably steer downstream generalization when inserting new associations, but struggles and has unpredictable effects when overriding existing associations, such as that between reward hacking and misalignment that produces EM. Our results suggest that, at the scales we test, SDF can make a model appear aligned with desired beliefs while steering its generalization from later training in unintended ways.
Sources
- Constitutional AI: Harmlessness from AI Feedback
- Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation
- Persona Vectors: Monitoring and Controlling Character Traits in Language Models
- Thought Crime: Backdoors and Emergent Misalignment in Reasoning Models
- Alignment faking in large language models
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- (Mis)generalization of Helpful-only Fine-tuning
- Agentic Misalignment: How LLMs Could Be Insider Threats
- Natural Emergent Misalignment from Reward Hacking in Production RL
- Removing Sandbagging in LLMs by Training with Weak Supervision
- Are Language Models Consequentialist or Deontological Moral Reasoners?
- AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors
- Believe It or Not: How Deeply do LLMs Believe Implanted Facts?
- Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time
- School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs
- Persona Features Control Emergent Misalignment
- Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignment
- ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection