Testing-Driven Reliability Audit of Trajectory-Based Early Outcome Prediction for LLM Agents: Target-Specific Calibration Transfer Persists Within a Single Benchmark
cs.AI
Submitted: 2026-09-22
Updated: 2026-09-22
Comments: 26 pages, 4 figures, 3 tables
Code: https://github.com/caoyanze426-crypto/early-outcome-calibration-audit
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- EarlyEval: Cheaper Agent Evaluation via Early Outcome Prediction
- AI Agents That Matter
- Doomed from the Start: Early Abort of LLM Agent Episodes via a Recall-Controlled Probe Cascade
- Last Step Matters: Early Uncertainty Cannot Predict Failure in Long-Horizon Agents
- Early Stopping for Large Reasoning Models via Confidence Dynamics
- On Calibration of Modern Neural Networks
- Online Reasoning Calibration: Test-Time Training Enables Generalizable Conformal LLM Reasoning
- CALIBER: Calibrating Confidence Before and After Reasoning in Language Models
- Maximum Likelihood with Bias-Corrected Calibration is Hard-To-Beat at Label Shift Adaptation
- Agentic Confidence Calibration
- When Calibration Rankings Reverse: Accuracy-Controlled Evaluation for Fair Comparison of LLMs
- A False Average: Pooled CoT-Monitor Accuracy Conceals a Reasoning-Dependent Fragility
- Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
- TIDE: Trajectory-based Diagnostic Evaluation of Test-Time Improvement in LLM Agents
- ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agents
- trajectory-judge: What Outcome-Only LLM Judges Miss on Agent Trajectories
- Counsel: A Meta-Evaluation Dataset for Agentic Tasks
- Confidence Comes from Experience: Experiential Confidence Estimation from Reasoning to Agents
- Do Agents Know When They Succeed? Calibrating Agent Confidence from Internal Representations
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection