On the Reliability of LLM-Based Vulnerability Patching Benchmarks

arXiv:2610.10150 · cs.CR, cs.SE · Submitted 2026-10-07 · Read on arXiv

cs.CR, cs.SE

Submitted: 2026-10-07

Updated: 2026-10-07

Code: https://github.com/ulikunitz/xz

Terminology

• Benchmark — 118× in this paper · explained in 2 episodes
• Trace — 16× in this paper · explained in 2 episodes
• Benchmarking — 8× in this paper · explained in 3 episodes
• Agent Harness — 5× in this paper · explained in 2 episodes
• Provenance — 5× in this paper · explained in 3 episodes
• Alignment — 4× in this paper · explained in 1 episode
• International Conference on Software Engineering — 4× in this paper
• Reinforcement Learning — 3× in this paper · explained in 4 episodes
• Ablation — 2× in this paper · explained in 1 episode
• Core — 2× in this paper · explained in 2 episodes
• Fine-Tuning — 2× in this paper · explained in 5 episodes
• Large Language Models (LLMs) — 2× in this paper · explained in 6 episodes
• AI Agents — 1× in this paper · explained in 2 episodes
• Adam — 1× in this paper · explained in 2 episodes
• Agentic Coding — 1× in this paper · explained in 1 episode
• Failure Taxonomy — 1× in this paper · explained in 2 episodes
• Fidelity — 1× in this paper · explained in 2 episodes
• Interpretability — 1× in this paper · explained in 15 episodes
• Pruning — 1× in this paper · explained in 4 episodes
• Reinforcement Learning (RL) — 1× in this paper · explained in 25 episodes
• Selection Bias — 1× in this paper · explained in 4 episodes
• Supervised Fine-Tuning (SFT) — 1× in this paper · explained in 8 episodes
• Systematic Bias — 1× in this paper · explained in 3 episodes

Sources

• Concerned with Data Contamination? Assessing Countermeasures in Code Language Model
• Large Language Models for Software Engineering: Survey and Open Problems
• Large Language Models for Software Engineering: A Systematic Literature Review
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks
• SoK: Towards Effective Automated Vulnerability Repair
• VADER: A Human-Evaluated Benchmark for Vulnerability Assessment, Detection, Explanation, and Remediation
• Agent Lightning: Train ANY AI Agents with Reinforcement Learning
• What's in a Benchmark? The Case of SWE-Bench in Automated Program Repair
• ARVO: Atlas of Reproducible Vulnerabilities for Open-Source Software
• Coding Agents with Multimodal Browsing are Generalist Problem Solvers
• Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study
• Agentless: Demystifying LLM-based Software Engineering Agents
• Benchmark Data Contamination of Large Language Models: A Survey
• A Survey of LLM-based Automated Program Repair: Taxonomies, Design Paradigms, and Applications
• SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
• Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving
• AutoCodeRover: Autonomous Program Improvement
• Fixing Security Vulnerabilities with AI in OSS-Fuzz
• Learning to Retrieve from Agent Trajectories

Related papers

• SoK: AI-Augmented Binary Reversing
• Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
• Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
• Efficient Fuzzy PSI under One-Sided Assumptions
• Sealing the Audit-Runtime Gap for LLM Skills
• Token Composition: A Graph Based on EVM Logs