What Does a Skill Actually Do? Estimands and Evaluation Validity for Tool and Skill Use in LLM Agents: A Critical Review

arXiv:2609.33153 · cs.SE, cs.AI · Submitted 2026-09-27 · Read on arXiv

cs.SE, cs.AI

Submitted: 2026-09-27

Updated: 2026-09-27

Terminology

• Benchmark — 31× in this paper · explained in 2 episodes
• Accuracy — 16× in this paper · explained in 1 episode
• Ablation — 13× in this paper · explained in 1 episode
• International Conference on Learning Representations — 13× in this paper
• Benchmarking — 11× in this paper · explained in 4 episodes
• Pruning — 8× in this paper · explained in 4 episodes
• Verification — 8× in this paper · explained in 2 episodes
• Actor — 7× in this paper · explained in 3 episodes
• Adaptation — 7× in this paper · explained in 1 episode
• AI Agents — 6× in this paper · explained in 2 episodes
• Exchangeability — 5× in this paper · explained in 2 episodes
• Reinforcement Learning — 5× in this paper · explained in 4 episodes
• Grounding — 4× in this paper · explained in 3 episodes
• Verifier — 4× in this paper · explained in 2 episodes
• Abstraction — 3× in this paper · explained in 1 episode
• Agent Architectures — 3× in this paper · explained in 1 episode
• Aggregation — 3× in this paper · explained in 1 episode
• Calibration — 3× in this paper · explained in 7 episodes
• Causal Inference — 3× in this paper · explained in 4 episodes
• Consistency — 3× in this paper · explained in 3 episodes
• Embeddings — 3× in this paper · explained in 2 episodes
• Abstention — 2× in this paper · explained in 1 episode
• Agent Skills — 2× in this paper · explained in 1 episode
• Agent Workflow Memory — 2× in this paper
• Core — 2× in this paper · explained in 2 episodes
• Deep Reinforcement Learning — 2× in this paper · explained in 2 episodes
• Generative AI — 2× in this paper · explained in 2 episodes
• Trace — 2× in this paper · explained in 2 episodes
• AUROC — 1× in this paper · explained in 2 episodes
• Action Space — 1× in this paper · explained in 1 episode
• Agent Memory — 1× in this paper · explained in 1 episode
• Agent Skill Induction — 1× in this paper
• Agentic AI — 1× in this paper · explained in 1 episode
• Agentic Evaluation — 1× in this paper · explained in 1 episode
• Agentic Task — 1× in this paper · explained in 1 episode
• Agentic Tasks — 1× in this paper · explained in 1 episode
• Amortization — 1× in this paper · explained in 1 episode
• Berkeley Function Calling Leaderboard — 1× in this paper
• Contextual Bandits — 1× in this paper · explained in 3 episodes
• Domain Shift — 1× in this paper · explained in 4 episodes
• Fine-Tuning — 1× in this paper · explained in 5 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Iterative Refinement — 1× in this paper · explained in 3 episodes
• Quantization — 1× in this paper · explained in 12 episodes
• Resource Constraints — 1× in this paper · explained in 3 episodes
• Retrieval-Invoked Actual-Use Effect — 1× in this paper
• Second Conference on Language Modeling — 1× in this paper
• Temporal Abstraction — 1× in this paper · explained in 2 episodes
• Unified Framework — 1× in this paper · explained in 5 episodes
• Visual Grounding — 1× in this paper · explained in 2 episodes

Sources

• AI Agents That Matter
• Voyager: An Open-Ended Embodied Agent with Large Language Models
• When Better Turns Do Not Make Better Agents: Diagnosing the Gap Between Next-Turn Metrics and Workflow Success
• ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
• TabAgent: A Framework for Replacing Agentic Generative Components with Tabular-Textual Classifiers
• Skill Following: Evaluating Actual Skill Use in Retrieval-Enabled LLM Agents
• Are Online Skill and Memory Modules Always Worth Their Tokens? A Budget-Constrained Study of Web Agents
• The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents
• SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks
• What Does an LLM-Agent Leaderboard Rank Actually Compare?
• From Relevance to Execution Utility: Reward-Aware Dynamic Execution Gating for Skill-Based LLM Agents
• More Skills, Worse Agents? Skill Shadowing Degrades Performance When Expanding Skill Libraries
• Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay
• The Replay Gap: Static Evaluation of Model Switching in LLM Agents Scores the Wrong World
• Accurate Failure Prediction in Agents Does Not Imply Effective Failure Prevention
• The Menu Is an Execution Prior: State-Path Tool Menus for Online Agents
• Skills Know Their Neighbors: Cluster-Contrastive Capability Pages for Skill Retrieval
• Direct and Indirect Effects

Related papers

• Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
• GitSkills: A Dataset of Agent Skills on GitHub
• SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces
• PackMonitor: Enabling Zero Package Hallucinations Through Decoding-Time Monitoring
• IntentCoding: Amplifying User Intent in Code Generation
• Incentives and Outcomes in Bug Bounties