Certified Selective Automation of LLM Agent Evaluation

arXiv:2609.34320 · cs.CL · Submitted 2026-09-28 · Read on arXiv

cs.CL

Submitted: 2026-09-28

Updated: 2026-09-28

Terminology

• GRPO — 39× in this paper · explained in 6 episodes
• AUROC — 14× in this paper · explained in 2 episodes
• Calibration — 11× in this paper · explained in 7 episodes
• Benchmark — 7× in this paper · explained in 2 episodes
• Pseudo-labels — 7× in this paper · explained in 2 episodes
• Clustering — 5× in this paper · explained in 2 episodes
• Accuracy — 4× in this paper · explained in 1 episode
• Reinforcement Learning — 3× in this paper · explained in 4 episodes
• Conformal Prediction — 2× in this paper · explained in 5 episodes
• Exchangeability — 2× in this paper · explained in 2 episodes
• LLM judge — 2× in this paper · explained in 2 episodes
• Adam — 1× in this paper · explained in 2 episodes
• Adaptation — 1× in this paper · explained in 1 episode
• Alignment — 1× in this paper · explained in 1 episode
• Benchmarking — 1× in this paper · explained in 4 episodes
• Fine-Tuning — 1× in this paper · explained in 5 episodes
• GRPO Objective — 1× in this paper · explained in 2 episodes
• LLM-as-a-Judge — 1× in this paper · explained in 3 episodes
• Provenance — 1× in this paper · explained in 4 episodes
• Selection Bias — 1× in this paper · explained in 4 episodes
• Self-Supervised Learning — 1× in this paper · explained in 8 episodes
• Semi-supervised Learning — 1× in this paper · explained in 4 episodes

Sources

• SCOPE: Selective Conformal Optimized Pairwise LLM Judging
• Qwen3-VL Technical Report
• $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
• AutoEval Done Right: Using Synthetic Data for Model Evaluation
• The BrowserGym Ecosystem for Web Agent Research
• WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?
• LoRA: Low-Rank Adaptation of Large Language Models
• Model-agnostic Selective Labeling with Provable Statistical Guarantees
• Language Models (Mostly) Know What They Know
• Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration
• AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories
• Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
• Autonomous Evaluation and Refinement of Digital Agents
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
• Self-Taught Evaluators
• An Illusion of Progress? Assessing the Current State of Web Agents
• Qwen3 Technical Report
• Self-Rewarding Language Models
• Agent-as-a-Judge: Evaluate Agents with Agents

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering