Safety Does Not Compose: Non-Decaying Loop State for Autonomous LLM Agents
cs.CR, cs.AI
Submitted: 2026-08-27
Updated: 2026-09-22
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large language model agents are increasingly deployed as autonomous loops.
Terminology
Abstract
Large language model agents are increasingly deployed as autonomous loops. Starting from one human goal, such a system repeatedly discovers work, plans, executes tool calls, verifies outcomes and persists state across many unattended iterations. The agent safeguards in wide use, however, are defined over a single trajectory, and their safety state is re-initialized when the next trajectory begins. We show that this is a failure of composition rather than an implementation detail. Our central result is a separation: against an attack whose evidence is fragmented across several iterations, every trajectory-scoped monitor has a true-positive rate equal to its false-positive rate, however expressive it is, because the evidence it would need never appears in the window it sees, whereas a monitor retaining cross-iteration state separates the two perfectly. We further show that the obvious repair of carrying a geometrically decaying risk score is insufficient, because the cooling-off period a patient adversary must wait is a constant that does not grow with the horizon N. We then present LoopHarness, which restores a persistent, non-decaying safety state at the loop level. Under mediated commits and an arbiter detection floor δ M, it bounds the expected number of unauthorized irreversible actions by B+m-1+m/δ M, a constant in N, of which the B+m-1 term is decided by a model-free rule and therefore survives a fully colluding verifier. We give a complete evaluation protocol on native Agent-SafetyBench tasks with paired clean and attacked episodes, an outer-state attack suite whose decisive evidence exists only across iterations, per-module ablations, and an adaptive white-box red team.
Sources
- Concrete Problems in AI Safety
- Design Patterns for Securing LLM Agents against Prompt Injections
- Measuring Progress on Scalable Oversight for Large Language Models
- Defeating Prompt Injections by Design
- Memory Injection Attacks on LLM Agents via Query-Only Interaction
- AI Control: Improving Safety Despite Intentional Subversion
- Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
- AI safety via debate
- Evaluating Language-Model Agents on Realistic Autonomous Tasks
- AI Safety Gridworlds
- SafeHarness: Lifecycle-Integrated Security Architecture for LLM-based Agent Deployment
- Testing Language Model Agents Safely in the Wild
- Ignore Previous Prompt: Attack Techniques For Language Models
- Toolformer: Language Models Can Teach Themselves to Use Tools
- Reflexion: Language Agents with Verbal Reinforcement Learning
- Voyager: An Open-Ended Embodied Agent with Large Language Models
- System-Level Defense against Indirect Prompt Injection Attacks: An Information Flow Control Perspective
- IsolateGPT: An Execution Isolation Architecture for LLM-Based Agentic Systems
- GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning
- Agent-SafetyBench: Evaluating the Safety of LLM Agents
Related papers
- SoK: AI-Augmented Binary Reversing
- Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
- Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
- Efficient Fuzzy PSI under One-Sided Assumptions
- Sealing the Audit-Runtime Gap for LLM Skills
- Token Composition: A Graph Based on EVM Logs