Toward Safe LLM Agents: A Survey of Specification, Verification, and Enforcement
cs.AI
Submitted: 2026-06-22
Updated: 2026-10-17
Comments: 37 pages, 2 figures, 11 tables. Revised version under review at ACM Computing Surveys. Data deposit: https://doi.org/10.5281/zenodo.23376335. PRISMA 2020 checklist included as an ancillary file
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large Language Model (LLM) agents take irreversible real-world actions, yet no system gives formally grounded, task-level safety guarantees for their plans.
Terminology
Abstract
Large Language Model (LLM) agents take irreversible real-world actions, yet no system gives formally grounded, task-level safety guarantees for their plans. We examined this gap in a PRISMA 2020 systematic review of six databases (20-29 May 2026) covering LLM agent specification, verification, and enforcement (2022-2026). We narratively synthesized 38 studies, assessed risk of bias per study, and graded evidence certainty with GRADE. First, specification is the bottleneck: natural-language-to-formal translation reaches only 24.8-34.2% semantic correctness on one Planning Domain Definition Language (PDDL) benchmark. Second, runtime monitoring is the most mature enforcement strategy, reducing unsafe actions by 65.37% in one household-agent study, without complete guarantees. Third, in one τ-Bench study of transactional tool-use agents, blocking 94% of unsafe actions left safe task completion below 5%, as agents found unsafe alternatives (the verifier tax). Fourth, no approach simultaneously achieves soundness, completeness, scalability, semantic grounding, and utility preservation. We contribute a three-level taxonomy and a ten-problem research agenda.
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems