LoopsBench: From Harness Engineering to Loop Engineering in Coding Agent Evaluation

arXiv:2608.00267 · cs.SE, cs.CL · Submitted 2026-08-10 · Read on arXiv

Han Li, Zhemin Fang, Rili Feng, Yingqi Zhao, Jiaheng Liu, Pengfei Gao, He Ye, Dayi Lin, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang

cs.SE, cs.CL

Submitted: 2026-08-10

Updated: 2026-08-11

Comments: Project page: https://loopsbench.ai/

Project page: https://loopsbench.ai

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

Trace — 68× in this paper · explained in 2 episodes
Benchmark — 40× in this paper · explained in 2 episodes
Resolve Rate — 24× in this paper
Verification — 10× in this paper · explained in 2 episodes
Fidelity — 5× in this paper · explained in 2 episodes
Benchmarking — 4× in this paper · explained in 3 episodes
Test Pass Rate — 3× in this paper
Width Ratio — 3× in this paper
Abstraction — 2× in this paper · explained in 1 episode
Alternation — 2× in this paper · explained in 1 episode
Generalization — 2× in this paper · explained in 13 episodes
Provenance — 2× in this paper · explained in 4 episodes
Robustness — 2× in this paper · explained in 5 episodes
AI Agents — 1× in this paper · explained in 2 episodes
Active Regression — 1× in this paper · explained in 1 episode
Agentic Coding — 1× in this paper · explained in 1 episode
Aggregation — 1× in this paper · explained in 1 episode
Consistency — 1× in this paper · explained in 3 episodes
Core — 1× in this paper · explained in 2 episodes
Reinforcement Learning — 1× in this paper · explained in 4 episodes
Topological Order — 1× in this paper · explained in 2 episodes
Verifier — 1× in this paper · explained in 2 episodes

Sources

Related papers