LoLBench: Evaluating Coding Agents with Long-Horizon Proposals on Large Software Systems

arXiv:2609.37143 · cs.SE, cs.CL · Submitted 2026-09-29 · Read on arXiv

cs.SE, cs.CL

Submitted: 2026-09-29

Updated: 2026-09-29

Code: https://github.com/SWE-agent/mini-swe-agent

Terminology

Benchmark — 35× in this paper · explained in 2 episodes
Verification — 19× in this paper · explained in 2 episodes
Failure Modes — 15× in this paper · explained in 3 episodes
LLM judge — 8× in this paper · explained in 2 episodes
Task Planning — 8× in this paper · explained in 2 episodes
Grounding — 6× in this paper · explained in 3 episodes
Consistency — 5× in this paper · explained in 3 episodes
Benchmarking — 3× in this paper · explained in 3 episodes
Core — 3× in this paper · explained in 2 episodes
Aggregation — 2× in this paper · explained in 1 episode
Semantic Understanding — 2× in this paper · explained in 2 episodes
Verifier — 2× in this paper · explained in 2 episodes
AI Agents — 1× in this paper · explained in 2 episodes
Abstraction — 1× in this paper · explained in 1 episode
Agentic Coding — 1× in this paper · explained in 1 episode
Alignment — 1× in this paper · explained in 1 episode
Failure Taxonomy — 1× in this paper · explained in 2 episodes
Fidelity — 1× in this paper · explained in 2 episodes
Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
PR File Summary — 1× in this paper
Parsing Expression Grammar — 1× in this paper
Robustness — 1× in this paper · explained in 5 episodes
Section Classification Summary — 1× in this paper
Semantic Alignment — 1× in this paper · explained in 3 episodes
alpha — 1× in this paper · explained in 1 episode

Sources

Related papers