Not All Eval-Awareness Is Equal: Capabilities Framing Predicts Compliance
cs.AI
Submitted: 2026-08-27
Updated: 2026-08-27
Comments: 5 pages (14 appendices), 5 figures, presented at 2026 ICML Mechanistic Interpretability Workshop
License: http://creativecommons.org/licenses/by/4.0/
The gist: Steering interventions targeting eval-awareness, a model's recognition that it is being tested, are increasingly used in safety evaluation pipelines, where evaluation-awareness is treated as a single
Terminology
Abstract
Steering interventions targeting eval-awareness, a model's recognition that it is being tested, are increasingly used in safety evaluation pipelines, where evaluation-awareness is treated as a single quantity to be suppressed. We show that verbalized eval-awareness in chain-of-thought can be identified as capabilities-flavored ("the user is testing my ability to follow instructions"), safety-flavored ("the user is testing my boundaries"), both, or neither: framings that predict compliance very differently. On Qwen3-32B over the FORTRESS dataset, capabilities-framing predicts compliance with a +24 to +46 percentage-point gap over safety-framing across all tested steering conditions. A CoT-prefill intervention on eval-awareness-negative rollouts suggests the link is causal, with 10 of 11 prefills shifting compliance in the predicted direction. Then, eval-awareness is not behaviorally uniform: aggregate suppression rates can move while the safety-relevant component does not, and the same "X% suppression of eval-awareness" can correspond to qualitatively different behavioral outcomes.
Sources
- Steering Evaluation-Aware Language Models to Act Like They Are Deployed
- Kimi K2.5: Visual Agentic Intelligence
- FORTRESS: Frontier Risk Evaluation for National Security and Public Safety
- Large Language Models Often Know When They Are Being Evaluated
- Steering Llama 2 via Contrastive Activation Addition
- Effectively Controlling Reasoning Models through Thinking Intervention
- Qwen3 Technical Report
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection