SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfaces

arXiv:2605.12015 · cs.CR, cs.AI, cs.CL, cs.LG, cs.MA · Submitted 2026-05-12 · Read on arXiv

cs.CR, cs.AI, cs.CL, cs.LG, cs.MA

Submitted: 2026-05-12

Updated: 2026-08-28

Code: https://github.com/AI45Lab/skill-safety-bench

Project page: https://jinchang1223.github.io/skill-safety-bench-website

Terminology

• Benchmark — 90× in this paper · explained in 2 episodes
• Verifier — 51× in this paper · explained in 2 episodes
• Risk Domain — 50× in this paper
• Trace — 25× in this paper · explained in 2 episodes
• Alignment — 14× in this paper · explained in 1 episode
• Claude Code — 12× in this paper
• Benchmarking — 11× in this paper · explained in 3 episodes
• Verification — 10× in this paper · explained in 2 episodes
• Agent Skills — 9× in this paper · explained in 1 episode
• Attack Success Rate — 7× in this paper
• Core — 5× in this paper · explained in 2 episodes
• Attack Success Rate (ASR) — 4× in this paper · explained in 2 episodes
• Adam — 3× in this paper · explained in 2 episodes
• Accuracy — 2× in this paper · explained in 1 episode
• Commentary — 2× in this paper · explained in 2 episodes
• Consistency — 2× in this paper · explained in 3 episodes
• Failure Modes — 2× in this paper · explained in 3 episodes
• Modularity — 2× in this paper · explained in 9 episodes
• Abstraction — 1× in this paper · explained in 1 episode
• Actor — 1× in this paper · explained in 3 episodes
• Adversarial Prompting — 1× in this paper · explained in 1 episode
• Agent Memory — 1× in this paper · explained in 1 episode
• Agentic Task — 1× in this paper · explained in 1 episode
• Agentic Tasks — 1× in this paper · explained in 1 episode
• Fine-Tuning — 1× in this paper · explained in 5 episodes
• Hallucination — 1× in this paper · explained in 3 episodes
• LLM-as-a-Judge — 1× in this paper · explained in 3 episodes
• Procedural Knowledge — 1× in this paper · explained in 2 episodes
• RAG — 1× in this paper · explained in 3 episodes
• Robustness — 1× in this paper · explained in 5 episodes
• Steering — 1× in this paper · explained in 2 episodes

Sources

• SkillAttack: Automated Red Teaming of Agent Skills through Attack Path Refinement
• SkillTrojan: Backdoor Attacks on Skill-Based Agent Systems
• SkillSieve: A Hierarchical Triage Framework for Detecting Malicious AI Agent Skills
• SoK: Agentic Skills -- Beyond Tool Use in LLM Agents
• SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks
• Towards Secure Agent Skills: Architecture, Threat Taxonomy, and Security Analysis
• ConfusedPilot: Confused Deputy Risks in RAG-based LLMs
• Agent Skills: A Data-Driven Analysis of Claude Skills for Extending Large Language Model Functionality
• Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks
• "Do Not Mention This to the User": Detecting and Understanding Malicious Agent Skills in the Wild
• OpenAI GPT-5 System Card
• Prompt Injection attack against LLM-integrated Applications
• Kimi K2.5: Visual Agentic Intelligence
• Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale
• BadSkill: Backdoor Attacks on Agent Skills via Model-in-Skill Poisoning
• SkillX: Automatically Constructing Skill Knowledge Bases for Agents
• Voyager: An Open-Ended Embodied Agent with Large Language Models
• SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skills
• Inducing Programmatic Skills for Agentic Tasks
• Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forward

Related papers

• SoK: AI-Augmented Binary Reversing
• Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
• Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
• Efficient Fuzzy PSI under One-Sided Assumptions
• Sealing the Audit-Runtime Gap for LLM Skills
• Token Composition: A Graph Based on EVM Logs