Cross-Layer Misalignment Detection in Agent Skills: A Progressive Loading-Aware Contrastive Learning Approach
Chengjun Zhang, Yang Gao, Jianna Hur, Jingjing Zhang, Sagar Samtani
cs.AI, cs.CR, cs.LG
Submitted: 2026-07-12
Comments: 10 pages, 5 pages supplemental. Accepted at the KDD 2026 Workshop on Evaluation and Trustworthiness of Agentic AI
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large language model (LLM) agents are increasingly extended through Agent Skills, reusable artifacts that package natural-language metadata, procedural instructions, and execution-time resources for
Terminology
Abstract
Large language model (LLM) agents are increasingly extended through Agent Skills, reusable artifacts that package natural-language metadata, procedural instructions, and execution-time resources for runtime use. As open-source skill marketplaces expand, users and agents increasingly rely on brief metadata to select third-party skills, making it difficult to detect inconsistencies between a skill's description and its true behavior, a problem we call cross-layer misalignment. To address this issue, we propose Progressive Loading-Aware Hierarchical Contrastive Learning (PL-HCL), an LLM-based framework that detects misalignment by modeling the layered structure of Agent Skills and learning cross-layer consistency. Using a normalized corpus of over 264,000 open-source skills and a human-verified challenge set, PL-HCL improves Macro-F1 from approximately 0.45 for unadapted baselines to 0.87-0.89 across evaluated LLM backbones. This approach offers an effective screening tool for users and operators, as well as design principles for detecting inconsistencies in layered digital artifacts.
Sources
- AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
- SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?
- SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfaces
- EMBER2024 -- A Benchmark Dataset for Holistic Evaluation of Malware Classifiers
- HiCLRE: A Hierarchical Contrastive Learning Framework for Distantly Supervised Relation Extraction
- SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks
- "Do Not Mention This to the User": Detecting and Understanding Malicious Agent Skills in the Wild
- Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale
- GAIA: a benchmark for General AI Assistants
- Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks
- Agents of Chaos
- Incorporating Hierarchy into Text Encoder: a Contrastive Learning Approach for Hierarchical Text Classification
- Hierarchical and Contrastive Representation Learning for Knowledge-aware Recommendation
- InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents
- Use All The Labels: A Hierarchical Multi-Label Contrastive Learning Framework
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection