Can Terminal Agents Trust Their Own Verification? Diagnosing and Improving Self-Verification

arXiv:2609.38812 · cs.CL, cs.LG · Submitted 2026-09-30 · Read on arXiv

cs.CL, cs.LG

Submitted: 2026-09-30

Updated: 2026-09-30

Code: https://github.com/harbor-framework/harbor

Terminology

• Verification — 218× in this paper · explained in 2 episodes
• Accuracy — 23× in this paper · explained in 1 episode
• Distillation — 15× in this paper · explained in 2 episodes
• Fine-Tuning — 5× in this paper · explained in 5 episodes
• Full-Trajectory Distillation — 5× in this paper
• Benchmark — 3× in this paper · explained in 2 episodes
• Calibration — 3× in this paper · explained in 7 episodes
• Consistency — 3× in this paper · explained in 3 episodes
• Reinforcement Learning — 3× in this paper · explained in 4 episodes
• Repair Success Rate — 3× in this paper
• Trace — 3× in this paper · explained in 2 episodes
• Verifier — 3× in this paper · explained in 2 episodes
• Aggregation — 2× in this paper · explained in 1 episode
• Generalization — 2× in this paper · explained in 13 episodes
• Generative AI — 2× in this paper · explained in 2 episodes
• LLM judge — 2× in this paper · explained in 2 episodes
• Student-Conditioned Verification Distillation — 2× in this paper
• Benchmarking — 1× in this paper · explained in 3 episodes
• Core — 1× in this paper · explained in 2 episodes
• Correct Pass Rate — 1× in this paper
• Error Detection Rate — 1× in this paper
• Error-Signal Precision — 1× in this paper
• Failure Modes — 1× in this paper · explained in 3 episodes
• Fidelity — 1× in this paper · explained in 2 episodes
• Iterative Refinement — 1× in this paper · explained in 3 episodes
• Regret — 1× in this paper · explained in 3 episodes
• Robustness — 1× in this paper · explained in 5 episodes
• Self-distillation — 1× in this paper · explained in 6 episodes
• Thinking mode — 1× in this paper · explained in 2 episodes
• Verification Outcome Rate — 1× in this paper
• Verification Pass Reliability — 1× in this paper
• Verification Trigger Rate — 1× in this paper

Sources

• FireAct: Toward Language Agent Fine-tuning
• Learning to Self-Verify Makes Language Models Better Reasoners
• Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents
• Independent Patch Verification for Coding Agents with a Bidirectional Reconstruct-and-Verify Framework
• Structured Agent Distillation for Large Language Model
• Student-Centered Distillation Narrows the Agentic Gap Between Small and Large LLMs
• S$^2$R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning
• Scaling Agentic Verifier for Competitive Coding
• Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
• SETA: Scaling Environments for Terminal Agents
• What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agents
• Beyond Fail-to-Pass: Iterative Hardening of Co-Generated Bug Reproduction Tests and Fixes
• SWE-Review: Closing the Loop on Issue Resolution with Agentic Code Review
• Large-Scale Terminal Agentic Trajectory Generation from Dockerized Environments
• What Makes Interaction Trajectories Effective for Training Terminal Agents?
• AgentTuning: Enabling Generalized Agent Abilities for LLMs
• GLM-5: from Vibe Coding to Agentic Engineering
• Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
• Failure as a Process: An Anatomy of CLI Coding Agent Trajectories
• Dream-RSI: Recursive Self-Improvement through Evolving Worlds

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering