What Does an Agentic Software Engineering Benchmark Measure? Profiling Task Demands and Agent Behaviour Beyond What Category Labels Reveal

arXiv:2609.01271 · cs.SE, cs.CL · Submitted 2026-09-01 · Read on arXiv

cs.SE, cs.CL

Submitted: 2026-09-01

Updated: 2026-09-01

Code: https://github.com/radinshayanfar/task_snc

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Benchmark — 112× in this paper · explained in 2 episodes
• Trace — 3× in this paper · explained in 2 episodes
• Agentic Coding — 2× in this paper · explained in 1 episode
• Benchmarking — 2× in this paper · explained in 3 episodes
• International Conference on Software Engineering — 2× in this paper
• Abstraction — 1× in this paper · explained in 1 episode
• Agentic Evaluation — 1× in this paper · explained in 1 episode
• Agentic System — 1× in this paper · explained in 2 episodes
• Core — 1× in this paper · explained in 2 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Scott-Knott Effect Size Difference — 1× in this paper
• Verifier — 1× in this paper · explained in 2 episodes
• World Model — 1× in this paper · explained in 7 episodes

Sources

• On Randomness in Agentic Evals
• FeatBench: Towards More Realistic Evaluation of Feature-level Code Generation
• Evaluating Large Language Models Trained on Code
• NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents
• Edit, But Verify: An Empirical Audit of Instructed Code-Editing Benchmarks
• Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation
• A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic System
• Agentic Software Issue Resolution with Large Language Models: A Survey
• The Rise of AI Teammates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering
• SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks
• SWE-smith: Scaling Data for Software Engineering Agents
• SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark
• When Elo Lies: Hidden Biases in Codeforces-Based Evaluation of Large Language Models

Related papers

• Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
• GitSkills: A Dataset of Agent Skills on GitHub
• SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces
• PackMonitor: Enabling Zero Package Hallucinations Through Decoding-Time Monitoring
• IntentCoding: Amplifying User Intent in Code Generation
• Incentives and Outcomes in Bug Bounties