On Evaluating and Improving Conversational Agents in Production

arXiv:2609.32092 · cs.MA, cs.AI, cs.CL, cs.ET, cs.IR · Submitted 2026-09-25 · Read on arXiv

cs.MA, cs.AI, cs.CL, cs.ET, cs.IR

Submitted: 2026-09-25

Updated: 2026-09-25

Terminology

• Benchmark — 7× in this paper · explained in 2 episodes
• Personalization — 7× in this paper · explained in 2 episodes
• LLM-as-a-Judge — 4× in this paper · explained in 3 episodes
• Multi-Agent System — 4× in this paper · explained in 5 episodes
• Accuracy — 2× in this paper · explained in 1 episode
• LLM judge — 2× in this paper · explained in 2 episodes
• Trace — 2× in this paper · explained in 2 episodes
• Agentic Evaluation — 1× in this paper · explained in 1 episode
• Benchmarking — 1× in this paper · explained in 3 episodes
• Clustering — 1× in this paper · explained in 2 episodes
• Core — 1× in this paper · explained in 2 episodes
• Embeddings — 1× in this paper · explained in 2 episodes
• Generative AI — 1× in this paper · explained in 2 episodes
• Iterative Refinement — 1× in this paper · explained in 3 episodes
• Reinforcement Learning — 1× in this paper · explained in 4 episodes

Sources

• GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents
• TREC CAsT 2019: The Conversational Assistance Track Overview
• Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines
• The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
• LLM Evaluators Recognize and Favor Their Own Generations
• Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations
• SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelines
• Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences
• Rethinking the Evaluation of Harness Evolution for Agents
• The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation
• The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search
• Self-Harness: Harnesses That Improve Themselves
• Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems
• CELEUS: Certifiable and Efficient LLM Evaluation via E-Processes
• How Reliable is Your Simulator? Analysis on the Limitations of Current LLM-based User Simulators for Conversational Recommendation
• RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulation

Related papers

• Highway Congestion Reduction through Reinforcement Learning Based Eulerian Headway Control
• You Only Align Once: Propagating Cooperative Behaviors in Multi-Agent Systems through Seed Agents
• Deny Without Disabling: Authorization-Paired Evaluation and Control for Multi-Agent Systems
• MA-SAPO: Multi-Agent Reasoning for Score-Aware Prompt Optimization
• PeroMAS: A Multi-agent System of Perovskite Material Discovery
• StitchCUDA: An Automated Multi-Agents End-to-End GPU Programing Framework with Rubric-based Agentic Reinforcement Learning