The Autonomy Tax: Defense Training Breaks LLM Agents

arXiv:2603.19423 · cs.CR, cs.AI, cs.LG · Submitted 2026-03-19 · Read on arXiv

cs.CR, cs.AI, cs.LG

Submitted: 2026-03-19

Updated: 2026-08-27

Terminology

• Benchmark — 15× in this paper · explained in 2 episodes
• Completion Rate — 10× in this paper
• Alignment — 8× in this paper · explained in 1 episode
• Failure Modes — 7× in this paper · explained in 3 episodes
• False Positive Rate — 5× in this paper
• Verification — 5× in this paper · explained in 2 episodes
• Fine-Tuning — 4× in this paper · explained in 5 episodes
• Trace — 4× in this paper · explained in 2 episodes
• Benchmarking — 3× in this paper · explained in 3 episodes
• Robustness — 3× in this paper · explained in 5 episodes
• Semantic Understanding — 3× in this paper · explained in 2 episodes
• Adversarial Attacks — 2× in this paper · explained in 1 episode
• Adversarial Examples — 2× in this paper · explained in 2 episodes
• Agent Harness — 2× in this paper · explained in 2 episodes
• Consistency — 2× in this paper · explained in 3 episodes
• Core — 2× in this paper · explained in 2 episodes
• Direct Preference Optimization — 2× in this paper
• Hallucination — 2× in this paper · explained in 3 episodes
• Large Language Model (LLM) — 2× in this paper · explained in 3 episodes
• True Positive Rate — 2× in this paper
• Adversarial Robustness — 1× in this paper · explained in 3 episodes
• Adversarial Training — 1× in this paper · explained in 1 episode
• Annotator Agreement — 1× in this paper · explained in 1 episode
• Causal Reasoning — 1× in this paper · explained in 2 episodes
• Diffusion Model — 1× in this paper · explained in 8 episodes
• Direct Preference Optimization (DPO) — 1× in this paper · explained in 9 episodes
• Distribution Shift — 1× in this paper · explained in 5 episodes
• Embeddings — 1× in this paper · explained in 2 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Generative AI — 1× in this paper · explained in 2 episodes
• Graceful Degradation — 1× in this paper · explained in 2 episodes
• Information Leakage — 1× in this paper · explained in 2 episodes
• Instruction Tuning — 1× in this paper · explained in 2 episodes
• Interpretability — 1× in this paper · explained in 15 episodes
• Multi-Agent System — 1× in this paper · explained in 6 episodes
• Positional Bias — 1× in this paper · explained in 2 episodes
• Redundancy — 1× in this paper · explained in 2 episodes
• Reinforcement Learning — 1× in this paper · explained in 4 episodes
• Representation Learning — 1× in this paper · explained in 2 episodes
• Selection Bias — 1× in this paper · explained in 4 episodes
• Transferability — 1× in this paper · explained in 4 episodes
• Transformer — 1× in this paper · explained in 2 episodes
• Transformer Architecture — 1× in this paper · explained in 2 episodes

Sources

• Constitutional AI: Harmlessness from AI Feedback
• SecAlign: Defending Against Prompt Injection with Preference Optimization
• Meta-SecAlign: Training LLMs against Prompt Injection for Robust Agents
• Generative AI in Transportation Planning: A Survey
• The Llama 3 Herd of Models
• Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
• Deliberative Alignment: Reasoning Enables Safer Language Models
• Baseline Defenses for Adversarial Attacks Against Aligned Language Models
• Mistral 7B
• "Someone Hid It": Query-Agnostic Black-Box Attacks on LLM-Based Retrieval
• A Personalized Conversational Benchmark: Towards Simulating Personalized Conversations
• Human-Aligned MLLM Judges for Fine-Grained Image Editing Evaluation: A Benchmark, Framework, and Analysis
• CMOOD: Concept-based Multi-label OOD Detection
• Defenses Against Prompt Attacks Learn Surface Heuristics
• StruQ: Defending Against Prompt Injection with Structured Queries
• Mitigating Hallucinations in Large Language Models via Causal Reasoning
• JailDAM: Jailbreak Detection with Adaptive Memory for Vision-Language Model
• Ignore Previous Prompt: Attack Techniques For Language Models
• Computational complexity of three-dimensional Ising spin glass: Lessons from D-Wave annealer
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

Related papers

• SoK: AI-Augmented Binary Reversing
• Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
• Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
• Efficient Fuzzy PSI under One-Sided Assumptions
• Sealing the Audit-Runtime Gap for LLM Skills
• Token Composition: A Graph Based on EVM Logs