ReproBench: Benchmarking LLM Agents on Reproducing Vulnerability From Scratch

arXiv:2609.34450 · cs.CR · Submitted 2026-09-28 · Read on arXiv

cs.CR

Submitted: 2026-09-28

Updated: 2026-09-28

Code: https://github.com/ReFirmLabs/binwalk

Terminology

Benchmark — 20× in this paper · explained in 2 episodes
Trace — 6× in this paper · explained in 2 episodes
Benchmarking — 5× in this paper · explained in 4 episodes
Verification — 5× in this paper · explained in 2 episodes
AI Agents — 4× in this paper · explained in 2 episodes
Network and Distributed System Security Symposium — 4× in this paper
Core — 3× in this paper · explained in 2 episodes
Agent Harness — 2× in this paper · explained in 2 episodes
Alignment — 2× in this paper · explained in 1 episode
Failure Taxonomy — 2× in this paper · explained in 2 episodes
Provenance — 2× in this paper · explained in 4 episodes
Security and Privacy — 2× in this paper
Agentic LLM Systems — 1× in this paper · explained in 1 episode
Clustering — 1× in this paper · explained in 2 episodes
Failure Modes — 1× in this paper · explained in 3 episodes
Large Language Model (LLM) — 1× in this paper · explained in 3 episodes
Phase-Specific Capability — 1× in this paper
Robustness — 1× in this paper · explained in 5 episodes

Sources

Related papers