Where Do LLMs Decide to Break the Rules? Mechanistic Localization of Prompt Injection Compliance

arXiv:2609.37737 · cs.CR · Submitted 2026-09-29 · Read on arXiv

cs.CR

Submitted: 2026-09-29

Updated: 2026-09-29

Terminology

• Benchmark — 16× in this paper · explained in 2 episodes
• AUROC — 9× in this paper · explained in 2 episodes
• Accuracy — 9× in this paper · explained in 1 episode
• Calibration — 5× in this paper · explained in 7 episodes
• Large Language Model — 5× in this paper
• Robustness — 5× in this paper · explained in 5 episodes
• Activation Patching — 4× in this paper · explained in 2 episodes
• Benchmarking — 4× in this paper · explained in 3 episodes
• Steering — 4× in this paper · explained in 2 episodes
• Generalization — 3× in this paper · explained in 13 episodes
• Representation Engineering — 3× in this paper · explained in 2 episodes
• Transformer — 3× in this paper · explained in 2 episodes
• Abstraction — 2× in this paper · explained in 1 episode
• Alignment — 2× in this paper · explained in 1 episode
• External Validation — 2× in this paper · explained in 2 episodes
• In International Conference on Learning Representations — 2× in this paper
• Instruction Tuning — 2× in this paper · explained in 2 episodes
• Interpretability — 2× in this paper · explained in 15 episodes
• Linear probe — 2× in this paper · explained in 2 episodes
• Activation Addition — 1× in this paper · explained in 1 episode
• Activation Steering — 1× in this paper · explained in 4 episodes
• Attention Mechanism — 1× in this paper · explained in 3 episodes
• Decode — 1× in this paper · explained in 2 episodes
• LLM judge — 1× in this paper · explained in 2 episodes
• Large Language Model (LLM) — 1× in this paper · explained in 3 episodes
• Latent Space — 1× in this paper · explained in 7 episodes
• Perplexity — 1× in this paper · explained in 3 episodes
• Provenance — 1× in this paper · explained in 4 episodes
• Regularization — 1× in this paper · explained in 3 episodes
• Trace — 1× in this paper · explained in 2 episodes

Sources

• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
• Eliciting Latent Predictions from Transformers with the Tuned Lens
• Localizing Model Behavior with Path Patching
• Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
• Mistral 7B
• Ignore Previous Prompt: Attack Techniques For Language Models
• Steering Language Models With Activation Engineering
• The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions
• Qwen3 Technical Report
• Representation Engineering: A Top-Down Approach to AI Transparency

Related papers

• SoK: AI-Augmented Binary Reversing
• Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
• Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
• Efficient Fuzzy PSI under One-Sided Assumptions
• Sealing the Audit-Runtime Gap for LLM Skills
• Token Composition: A Graph Based on EVM Logs