Beyond Attack-Success Rate: Action-Graded Severity Scale for Tool-Using AI Agents

arXiv:2607.07474 · cs.CR, cs.AI, cs.CL · Submitted 2026-07-08 · Read on arXiv

Harry Owiredu-Ashley

cs.CR, cs.AI, cs.CL

Submitted: 2026-07-08

Comments: 8 pages, 6 figures. Code and artifacts: https://github.com/Harry-Ashley/action-graded-severity

Code: https://github.com/Harry-Ashley/action-graded-severity

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Benchmark — 28× in this paper · explained in 2 episodes
• Trace — 14× in this paper · explained in 2 episodes
• alpha — 3× in this paper · explained in 1 episode
• AI Agents — 2× in this paper · explained in 2 episodes
• Benchmarking — 2× in this paper · explained in 3 episodes
• LLM judge — 2× in this paper · explained in 2 episodes
• Accountability — 1× in this paper · explained in 2 episodes
• Agent Interactions — 1× in this paper · explained in 1 episode
• Agentic AI — 1× in this paper · explained in 1 episode
• Calibration — 1× in this paper · explained in 7 episodes
• International Conference on Learning Representations — 1× in this paper
• LLM-as-a-Judge — 1× in this paper · explained in 3 episodes
• Robustness — 1× in this paper · explained in 5 episodes

Sources

• Ignore Previous Prompt: Attack Techniques For Language Models
• Agent-SafetyBench: Evaluating the Safety of LLM Agents
• Defending Against Indirect Prompt Injection Attacks With Spotlighting

Related papers

• SoK: AI-Augmented Binary Reversing
• Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
• Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
• Efficient Fuzzy PSI under One-Sided Assumptions
• Sealing the Audit-Runtime Gap for LLM Skills
• Token Composition: A Graph Based on EVM Logs