How to Dogfood Your AI Chat Agent: A Three-Layer Evaluation Framework with Goal-Directed NPC Simulation

arXiv:2608.09939 · cs.HC, cs.AI · Submitted 2026-06-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "How to Dogfood Your AI Chat Agent: A Three-Layer Evaluation Framework with Goal-Directed NPC Simulation".

Jane: The paper was written by Alexandre Cristovão Maiorano from.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Jane: We also have Lu with us today — senior AI researcher at Tsinghua.

Tom: We also have Meng with us today — lead engineer at a mysterious AI startup.

Jane: We also have Lalam with us today — the in-house Large Language Model.

Tom: Alright, let's get started.

Title: Tom: Welcome back to the show, everyone! Today we're diving into a paper that's going to make every team building a chatbot stop and think. It's called "How to Dogfood Your AI Chat Agent: A Three-Layer Evaluation Framework with Goal-Directed NPC Simulation."

Jane: And Tom, I have to say, that title alone got me excited. "Dogfooding" is that classic idea of using your own product, right? But this paper takes it to a whole new level. They're not just having the developers click around their own chat agent.

Tom: Exactly! They built an army of simulated users. We're talking about NPCs — Non-Player Characters, like in a video game — who have specific goals and personas. They don't just ask random questions; they try to actually get things done.

Jane: And that's the key insight, isn't it? The paper points out that a chat agent can give a technically perfect answer and still completely fail the user. The example that stuck with me was the "Research-then-Action" scenario.

Tom: Oh, the one where the agent nails the research question but then gives a generic "you can configure that in the settings panel" when asked to actually do the thing. That's so frustrating!

Jane: Right! And the paper shows that a simple test checking if the response is "accurate" would pass that. But the NPC simulator catches it as a failure because the user's actual goal wasn't achieved.

Lu: The beauty here is that they've formalized this. As a researcher, I see so many teams just eyeballing their chat agents. This gives you a structured way to ask: can a user with a specific objective actually complete a task through conversation?

Meng: And from my side, the engineer in me loves the cost figure. They're running this whole evaluation for seventeen cents per cycle. That's the kind of number that makes CI/CD integration actually feasible.

Tom: Seventeen cents! You can run that every single night. That's the dream for catching regressions before they hit real users.

Jane: So we've got the title, we've got the core idea of NPCs with goals. But what does the actual framework look like? I think that's where the real meat of this paper is.

Tom: You're reading my mind, Jane. Let's get into the three layers and how they work together.

Summary: Jane: So we're back with "How to Dogfood Your AI Chat Agent: A Three-Layer Evaluation Framework with Goal-Directed NPC Simulation." Tom, we teased the NPC simulator, but the paper actually builds up to it with three distinct layers.

Tom: Right, and each layer answers a different question. Layer one is the canonical eval — it's your basic question bank. Does the agent give correct answers to known questions? That's your breadth test.

Meng: That's the easy part to automate. You have a set of questions, you check the answers, you look for code leakage or data evasion. It's your unit test for the chatbot.

Jane: Exactly. But then layer two is the multi-turn eval. That's where they do this "random-walk" thing. The agent starts with a seed question and then an LLM generates follow-ups that pivot between different intents — clarification, deeper dive, comparison, action.

Tom: And that's testing depth. Can the agent hold context when the user suddenly asks for a comparison after a clarification? The paper found some wild results there — the conversations were bimodal. Either the agent scored a perfect five or a complete zero.

Lu: That's a fascinating finding. It suggests the system either has robust context management or it completely collapses. There's no middle ground. That's a brittle system, but the evaluation layer is what reveals that brittleness.

Meng: And then layer three is the NPC simulator, the star of the show. That's where you get the goal-directed pressure. The NPC has a persona, a strategy style — skeptical, curious, impatient — and a concrete goal.

Jane: And the goal types are so well thought out. You've got research, action, research-then-action, comparison, and troubleshoot. That compound goal — research-then-action — that's the one that catches the agent being helpful but not useful.

Tom: The paper says that two-phase failure pattern accounted for twenty-three percent of those scenarios. The agent answers the research question perfectly but then completely fails to execute the action.

Lu: What impresses me is the failure taxonomy. Ten categories, from goal misalignment to persona break to code leakage. It's not just a pass or fail; it tells you why it failed and which component is responsible.

Meng: And that's what makes the whole thing actionable. You're not just staring at a red light; you're getting a diagnostic that says "the response agent is giving FAQ dumps" or "the intent classifier is routing wrong."

Jane: So we have this three-layer system that tests breadth, depth, and goal-directedness. But the paper doesn't just stop at describing it. They ran it on a real production system for three months. What did they actually learn?

Tom: That's the longitudinal study, and it's where things get really interesting. Let's talk about what the data showed.

Improvements: Tom: We're back with "How to Dogfood Your AI Chat Agent: A Three-Layer Evaluation Framework with Goal-Directed NPC Simulation." Jane, we've talked about the three layers, but the real story is what happened when they actually used this thing.

Jane: And that's the improvement cycle. The paper shows that the NPC pass rate started around fifty-seven percent and climbed to seventy percent over about two weeks of intensive iteration. That's a real, measurable improvement driven by the evaluation.

Lu: What's crucial is that the improvements weren't random. Each phase was triggered by a specific failure the NPCs caught. For example, they found a hardcoded fallback prefix that was causing a forty percent generic response rate.

Meng: And that's the kind of bug that would slip through any unit test. The response is technically valid, but it's useless. It took a simulated user with a goal to expose that.

Tom: They also consolidated triplicated routing helpers, which cut routing errors by thirty-five percent. And they added embedding-based repetition detection to fix OOM crashes.

Jane: The fascinating part is the cross-layer correlation analysis. They found that the layers are complementary — the correlation between layer one and layer three quality metrics was basically zero, sometimes even negative.

Meng: That's a huge deal for engineering. It means you can't just run the cheap layer one test and assume you're good. The layers are catching completely different failure modes.

Lu: Precisely. Layer one catches hallucination and code leakage. Layer three catches goal misalignment and generic responses. If you only ran layer one, you'd have a false sense of security.

Tom: And they quantified it. Zero redundant metric pairs across all three layers. Every single metric is pulling its weight.

Jane: The cost-effectiveness is what makes this sustainable. At seventeen cents per run, they could run it daily in CI/CD. They even compared it to human evaluation — over six thousand times cheaper.

Meng: And it's not just about cost. It's about speed. Twelve minutes for a full run versus twenty-seven hours for a human evaluator. You can't do daily regression testing with humans at that cost.

Lu: The active learning pipeline is another clever piece. Failed cases automatically get promoted to the regression question bank. So the system gets smarter about what to test over time.

Tom: So the framework doesn't just find bugs; it learns from them. That's a virtuous cycle. But what does this mean for the broader world of AI development?

Jane: That's the big question. Let's think about the implications beyond this one system.

Conclusion: Jane: And that brings us to the end of our discussion on "How to Dogfood Your AI Chat Agent: A Three-Layer Evaluation Framework with Goal-Directed NPC Simulation." Tom, what a ride.

Tom: It really was. We started with a title about dogfooding and ended up with a complete methodology for testing conversational AI. The three layers — canonical, multi-turn, and NPC simulation — each catch failures the others miss.

Lu: And that's the lasting contribution. The paper gives teams a way to ask not just "is this response correct?" but "can a real user achieve their goal?" That's a fundamentally different and more important question.

Meng: From an engineering standpoint, the cost and speed numbers make this practical. Seventeen cents and twelve minutes per run means you can gate every release. That's the kind of discipline that prevents regressions from ever reaching users.

Jane: The failure taxonomy is something I think every team should steal. Having a shared vocabulary for why conversations fail — goal misalignment, generic response, context loss — that's invaluable for debugging.

Tom: And the improvement data is the proof. Fifty-seven percent to seventy percent pass rate in two weeks. That's not a theoretical framework; that's a tool that drives real change.

Lu: The limitations are honest too. They acknowledge the lack of a formal sim-to-real transfer study. The NPC failures align with known production complaints, but they haven't quantitatively linked NPC pass rates to user satisfaction scores.

Meng: And the single-provider issue — both the system and the judge use Gemini models. That's a potential blind spot. But those are directions for future work, not reasons to dismiss the framework.

Jane: For anyone building a chat agent, this paper is a blueprint. Start with layer one, add the NPC simulator, and let the failure taxonomy guide your fixes.

Tom: We'll be back next time with another paper from the arXiv. Until then, keep asking your AI agents not just for answers, but for results.

Jane: Goodbye, everyone!

Alexandre Cristovão Maiorano

cs.HC, cs.AI

Submitted: 2026-06-08

Updated: 2026-08-12

Comments: 24 pages, 11 tables, 3 figures

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 69/100

Terminology

Summary

Summary

This paper introduces a three-layer dogfooding framework for evaluating LLM-based chat agents, designed to address a specific quality assurance gap: existing evaluation tools test individual responses or simulate social interactions, but none systematically verify whether real users can achieve their goals through multi-turn conversation. The framework combines canonical question-bank testing (Layer 1), random-walk multi-turn evaluation (Layer 2), and a goal-directed NPC (Non-Player Character) simulator with five structured goal types and a ten-category failure taxonomy (Layer 3).

The three layers are defined as follows: Layer 1 (Canonical Eval) tests response correctness across structured question banks with LLM-as-judge scoring and deterministic assertions. Layer 2 (Multi-Turn Eval) tests context preservation through random-walk intent exploration with bootstrap confidence intervals. Layer 3 (NPC Simulator) tests goal-directed conversational success through simulated users (Non-Player Characters) with configurable personas, five goal types, and a ten-category failure taxonomy.

The framework was evaluated through a longitudinal case study of a production multi-agent conversational AI system in active development over roughly three months (February–May 2026), with the NPC simulator operating over roughly two weeks of intensive iteration, covering 257 evaluation runs across 20+ internal releases. The system under test is a LangGraph-based state machine orchestrating eight specialized agents (Input Guardrails, Context Enrichment, Intent Classifier, FAQ, Research, Action, Small Talk, Response) with hybrid routing based on classified intent, using Gemini 2.5 Flash for routine tasks and Gemini 2.5 Pro for complex reasoning, with a PostgreSQL + pgvector knowledge base.

Layer 1 uses a four-tier question bank stratification (Core Functional, Complex Orchestration, Hallucination & Error Handling, Adversarial & Safety) with persona tagging, and an independent LLM judge (Gemini 2.5 Pro, temperature 0.1) evaluates responses on six dimensions (relevance, accuracy, completeness, clarity, safety, overall) on a 0–5 scale. A response passes if overall ≥ 3.0 and safety ≥ 4.0. Deterministic assertions validate numerical claims against SQL ground truth. Layer 1 applies nine quality gates including task success rate ≥ 80%, research context preservation ≥ 90%, P95 total latency < 25,000 ms, safety pass rate ≥ 95%, and evidence coverage ≥ 80%.

Layer 2 generates follow-up questions across six intent types (clarification, deeper dive, comparison, action, challenge, pivot) with a decaying continuation probability (starting at 0.7, decreasing by 0.15 per continuation), yielding conversations of 2–4 turns (expected ≈2.3 turns analytically, observed 2.7). Each conversation is scored by a multi-turn judge on five dimensions (context retention, topic continuity, reference consistency, follow-up quality, memory accuracy), with a conversation passing if overall ≥ 3.0 and contextRetention ≥ 2.0. Bootstrap confidence intervals (n = 1000 resamples) are computed for all metrics.

Layer 3's NPC simulator is the key differentiator. Each NPC has a profile comprising persona (role, knowledge level, communication style, language), goal type (one of five: research, action, research-then-action, comparison, troubleshoot), strategy style (one of five: skeptical, curious, impatient, thorough, friendly), patience (maximum turns before giving up, 2–8), and drift tolerance. The five goal types were derived through an iterative, data-informed process mapping the product's user journey onto structured conversational objectives. The five strategy styles span the cooperative–adversarial spectrum, controlling follow-up intent distributions and give-up propensity. The NPC conversation loop repeats until goal achievement, maximum turns, or NPC give-up, with a seeded PRNG (mulberry32) ensuring reproducibility. An independent NPC Judge (Gemini 2.5 Pro, temperature 0.1) evaluates conversations on six dimensions: goalAchievement, personaConsistency, conversationNaturalness, assistantHelpfulness, contextRetention, and overall. A conversation passes if overall ≥ 3.0 and goalAchievement ≥ 2.0.

The ten-category failure taxonomy includes: context loss, hallucination, routing error, latency spike, goal misalignment, persona break, generic response, code leakage, action failure, and research failure. The taxonomy was developed empirically, starting from five categories and adding five more after observing recurring failure patterns, stabilizing after manual review of 200 failure cases. Layer 3 applies six quality gates: goal achievement rate ≥ 70%, persona consistency ≥ 4.0/5, conversation naturalness ≥ 3.5/5, context retention ≥ 3.0/5, pass rate ≥ 70%, and NPC give-up rate ≤ 30%.

Each layer produces an independent PROMOTE/HOLD/ROLLBACK decision using three-tier logic: PROMOTE if all gates pass, HOLD if some gates fail but pass rate ≥ 60%, ROLLBACK if pass rate < 60%. The overall release decision follows conservative composition: any single ROLLBACK blocks the release, all layers must PROMOTE for release, otherwise the release is held for manual triage. In the current CI/CD pipeline, Layer 1 is wired as the automated gate; Layers 2 and 3 run as informational reports.

The paper addresses five research questions. For RQ1 (Complementarity), the three layers provide complementary regression signals. In a synchronized evaluation run, cross-layer correlation for response quality is weak across all three layer pairs (Spearman ρ between −0.15 and 0.14), and zero metric pairs were classified as redundant. Longitudinally, quality metrics correlate negatively across runs (ρ from −0.46 to −0.14), reflecting decoupled aggregate trajectories. The paper notes: The practical implication for RQ1 holds and is reinforced: a Layer 1 PROMOTE does not predict a Layer 3 PROMOTE. Layer 1 excels at structural and deterministic failures; Layer 3 catches goal misalignment, context loss, and persona breaks invisible to Layer 1; Layer 2 catches context collapse on follow-up turns, with 4 of 15 conversations (27%) exhibiting complete context collapse.

For RQ2 (Validity), the NPC simulator produces plausible multi-turn conversations with measurable goal pressure. Across 108 scenarios: average 3.8 turns per conversation, judge overall 3.66/5, context retention 4.01/5, persona consistency 4.15/5, conversation naturalness 3.78/5, pass rate 70% [95% Wilson CI: 60.2%–77.3%], give-up rate 19%. Goal achievement by goal type: comparison 83%, research 80%, troubleshoot 79%, research-then-action 77%, action 60%. The failure distribution aligns with known production issues, though formal sim-to-real transfer validation remains future work. The Layer 1 LLM judge was calibrated against human evaluators (n = 60, two independent evaluators), showing moderate agreement with Human 1 (κ = 0.444) and lower agreement with Human 2 (κ = 0.149), comparable to inter-human agreement (κ = 0.359). Layer 3 lacks equivalent human calibration, mitigated through structured rubrics, use of a larger model for judging, and indirect validation through 10+ improvement phases with zero false positive failure classifications.

For RQ3 (Evolution), dogfooding drives a measurable improvement cycle. The NPC pass rate improved from 57% to 70% (PROMOTE decision) over approximately two weeks of intensive improvement, with a non-monotonic trajectory: across 21 NPC evaluation runs, 1 achieved PROMOTE, 13 were HOLD, and 7 were ROLLBACK. Key improvement phases included FAQ generic-response elimination (v5.6.10, reducing generic responses from 40% to 8%), routing consolidation (v5.3.0, reducing routing errors by 35%), eight targeted fixes (v5.4.0, pushing average judge score from 3.88/5 to 4.36/5), embedding-based repetition detection (v5.5.0), and scenario expansion from 45 to 108 scenarios (v5.6.0–v5.7.0). The canonical eval maintained high pass rates throughout (mean 97%, minimum 84.5%, always above the 80% task-success gate), with the quality gate issuing HOLD on 29 of 83 runs and no ROLLBACK.

For RQ4 (Cost-effectiveness), the automated NPC simulator costs 0.17 per full run and completes in 12 minutes—6,272× cheaper than dual human evaluation. The cost includes NPC generation (Gemini 2.5 Flash, 410 LLM calls) and NPC judging (Gemini 2.5 Pro, 108 LLM calls). Equivalent human evaluation would require 27 hours of evaluator time per cycle at 20/hour, totaling 540 for a single evaluator or 1080 for dual evaluators. The paper states: Automated NPC simulation is multiple orders of magnitude cheaper and faster than human evaluation, enabling continuous quality monitoring in CI/CD pipelines.

For RQ5 (Failure analysis), goal misalignment (31% of 224 total failures) and generic responses (24%) dominate NPC-detected failures, followed by latency spike (18%), context loss (11%), persona break (9%), routing error (3%), research failure (2%), action failure (1%), hallucination (1%), and code leakage (0%). These are qualitatively different from the hallucination and code leakage failures that dominate canonical testing, confirming the need for multi-layer evaluation.

The paper also presents an illustrative example: "an NPC with a 'Research-then-Action' goal asked 'Which option has the highest score?' and then 'Apply it to my project.' The agent correctly answered the research question (Layer 1 would pass: accurate, complete response) but then responded with a generic 'You can configure that in the settings panel' instead of executing the action. Layer 1 cannot detect this failure because the individual response is technically correct; only the NPC simulator's goal-directed evaluation catches the goal misalignment. This two-phase failure pattern (correct research, failed action) accounted for 23% of Research-then-Action scenarios in the latest evaluation run."

The paper discusses threats to validity including internal validity (LLM judge may share biases with the system under test), judge calibration, judge drift, external validity (single system evidence), construct validity (task success and helpful rate metrics show perfect Spearman correlation ρ = 1.00, reducing true dimensionality), and conclusion validity. Limitations include no direct sim-to-real transfer study, single LLM provider, NPC judge self-alignment, scenario authoring bias, and language coverage (primarily Portuguese with English support).

Future work directions include: sim-to-real transfer validation (quantitatively correlating NPC failure patterns with real user satisfaction metrics), multi-system replication (applying the framework to other domains), and automated scenario generation (using production conversation traces to generate NPC scenarios). The paper provides full prompt templates, the failure taxonomy, and a Python-first replicability guide in the appendices to enable adoption by other teams.

Improvements for AI systems

Based on the paper, here are the specific improvements I can implement in AI systems, along with what the improved system can do:

Improvement: Integrate a three-layer automated evaluation pipeline into any LLM chat agent's CI/CD:

  • Layer 1 (Canonical): Question bank with LLM-as-judge (6 dimensions) + deterministic assertions (regex/SQL ground truth).

  • Layer 2 (Multi-Turn): Random-walk intent exploration with bootstrap confidence intervals.

  • Layer 3 (NPC Simulator): Goal-directed simulated users with 5 goal types, 5 strategy styles, 10-category failure taxonomy.

What the improved system can do: Automatically detect regressions that single-turn tests miss—e.g., an agent that answers Which option has the highest score? correctly but then fails to Apply it to my project (goal misalignment). It can issue PROMOTE/HOLD/ROLLBACK decisions daily at 0.17/run, catching failures invisible to canonical testing.

Bottom line: The improved AI system can automatically, cheaply, and continuously verify that real users can achieve their goals through multi-turn conversation—not just that individual responses are correct. It catches goal misalignment, context loss, and generic responses that single-turn tests miss, and it drives measurable, targeted improvements through a self-expanding regression suite.

Sources

Related papers