From Offline Proxies to Online Decisions: A Layered Engagement Evaluation Framework for Conversational AI

arXiv:2609.25408 · cs.AI, cs.IR, cs.LG · Submitted 2026-09-21 · Read on arXiv

cs.AI, cs.IR, cs.LG

Submitted: 2026-09-21

Updated: 2026-09-21

Comments: 11 pages main text, 9 pages supplementary material; 2 figures, 25 tables

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Calibration — 43× in this paper · explained in 7 episodes
• Abstention — 14× in this paper · explained in 1 episode
• Ablation — 12× in this paper · explained in 1 episode
• Alignment — 11× in this paper · explained in 1 episode
• Robustness — 7× in this paper · explained in 5 episodes
• Aggregation — 6× in this paper · explained in 1 episode
• Provenance — 5× in this paper · explained in 4 episodes
• Clustering — 3× in this paper · explained in 2 episodes
• Consistency — 3× in this paper · explained in 3 episodes
• Generalization — 3× in this paper · explained in 13 episodes
• Information Gain — 3× in this paper · explained in 2 episodes
• Accuracy — 2× in this paper · explained in 1 episode
• LLM judge — 2× in this paper · explained in 2 episodes
• Reinforcement Learning — 2× in this paper · explained in 4 episodes
• AI Agents — 1× in this paper · explained in 2 episodes
• Abstention mechanism — 1× in this paper · explained in 1 episode
• Actionability — 1× in this paper · explained in 3 episodes
• Agentic System — 1× in this paper · explained in 2 episodes
• Core — 1× in this paper · explained in 2 episodes
• Data Provenance — 1× in this paper · explained in 2 episodes
• Deep Reinforcement Learning — 1× in this paper · explained in 2 episodes
• Exposure Bias — 1× in this paper · explained in 2 episodes
• Grounding — 1× in this paper · explained in 3 episodes
• LLM-as-a-Judge — 1× in this paper · explained in 3 episodes
• Scaling Laws — 1× in this paper · explained in 4 episodes
• Selection Bias — 1× in this paper · explained in 4 episodes

Sources

• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• ForTune: Running Offline Scenarios to Estimate Impact on Business Metrics
• Large Language Models as User-Agents for Evaluating Task-Oriented-Dialogue Systems
• Do Offline Metrics Predict Online Performance in Recommender Systems?
• Can LLM Agents Simulate Multi-Turn Human Behavior? Evidence from Real Online Customer Behavior Data
• Choosing a Proxy Metric from Past Experiments
• Evaluating the Surrogate Index as a Decision-Making Tool Using 200 A/B Tests at Netflix

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
• VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection