ATOD: An Evaluation Framework and Benchmark for Agentic Task-Oriented Dialogue Systems

Recent advances in task-oriented dialogue (TOD) systems, driven by large language models (LLMs) with extensive API and tool integration, have enabled conversational agents to coordinate interleaved goals, maintain long-horizon context, and act proactively through asynchronous execution. This paper introduces ATOD as a benchmark dataset and ATOD-Eval as a holistic evaluation framework designed to systematically assess these advanced agentic behaviors in TOD systems.

ATOD: The Benchmark Dataset

ATOD is a synthetic dataset and generation pipeline designed to capture key characteristics of Advanced TOD, including multigoal coordination, dependency management, memory, adaptability, and proactivity. The dataset is constructed through a modular LLM-driven pipeline involving several stages:

1. Co-occurrence Graph Construction and Goal Trajectory Sampling: This involves constructing a goal co-occurrence graph from an underlying dialogue dataset where each node represents a unique goal and weighted edges reflect empirical co-occurrence frequency. Candidate goal sets are then sampled via stratified random walks of varying lengths over G, preserving realistic correlations while introducing diversity.

2. Annotation of Goal Trajectories and Complexity Categorization: An LLM annotates slot values, inter-goal dependencies DS capturing prerequisite or blocking relations, and natural-language goal descriptions. Each trajectory is assigned a complexity label based on quantitative attributes (e.g., number of goals, dependency density) and qualitative factors (e.g., interleaving or opportunities for proactivity).

3. Dialogue Generation: The synthesis is conditioned on the annotated trajectory τ = (S, DS, c(S)), where the LLM produces a natural multi-turn conversation that realizes the specified goals while exhibiting interleaving, asynchronous execution, proactive assistance, and dependency-aware coordination.

4. Turn-level Goal Status Annotation: An LLM annotator performs iterative turn-level analysis to label the status of each goal at every dialogue turn, providing a rich reference for benchmarking multi-goal tracking and asynchronous or interleaved progressions.

ATOD-Eval: The Evaluation Framework

Building on ATOD, ATOD-Eval is proposed as a holistic evaluation framework that translates the advanced TOD dimensions into fine-grained metrics and supports reproducible offline and online evaluation. This framework unifies evaluation by jointly assessing five key dimensions: goal completion, dependency management, memory consistency, adaptability, proactivity, and multigoal coordination. It utilizes an agentic memory system to evaluate models directly on dialogue text by assessing whether they can consistently maintain and update goal trajectories throughout interaction.

Key Evaluation Metrics

ATOD-Eval defines metrics across three dimensions:

1. Task Completion and Efficiency: This includes the Dependency-Aware Goal Completion Rate (dGCR), which considers only goals whose prerequisites are satisfied, avoiding bias from dependency-locked goals. It also measures Turns to Completion (NTC) for execution efficiency.

2. Agentic Capability Metrics: These assess behaviors such as Memory Recall Accuracy and Proactivity Effectiveness, which evaluate goal or state changes initiated without explicit user prompts and their contextual appropriateness.

3. Response Quality Metrics: These focus on turn-level relevance and dialogue-level coherence, ensuring systems maintain natural and consistent interactions alongside effective goal management.

Agentic Memory System

The evaluation backbone incorporates an agentic memory system consisting of a dual memory store: (i) a structured goal database Dsym, which persistently records symbolic metadata like status history; and (ii) a semantic vector store Dvec, which indexes embeddings for similarity-based retrieval. The turn-level processing pipeline applies four stages to maintain the lifecycle of all goals: (i) goal extraction from the current utterance and context; (ii) existence checking against the dual memory store; (iii) updating or inserting goals with dependency evolution; and (iv) proactive auditing to keep active states consistent.

Experimental Validation

Experiments validate that ATOD-Eval enables comprehensive assessment across task completion, agentic capability, and response quality. The proposed evaluator is shown to consistently outperform competitive baselines under this evaluation setting when assessed on goal detection and status tracking, while simultaneously achieving lower per-turn update latency and token usage. Furthermore, correlation analysis shows that Memory Recall Accuracy correlates most strongly with dGCR in both settings, highlighting the critical role of accurate memory in dependency-aware success. The efficiency analysis demonstrates that the method achieves the lowest per-turn update latency and consumes fewer tokens compared to baselines like LLM-Rsum.

Conclusion

ATOD and ATOD-Eval provide a unified and scalable foundation for evaluating next-generation TOD systems. The framework successfully addresses the gap in existing benchmarks by providing a purpose-built dataset that explicitly encodes complex agentic behaviors, enabling consistent assessment across static datasets and real-time deployments.

The gist: ATOD is a benchmark dataset and ATOD-Eval is a holistic evaluation framework designed to systematically assess advanced agentic behaviors in task-oriented dialogue systems.

Here are the specific improvements that can be made to current AI systems by implementing the ATOD/ATOD-Eval framework, along with what these improved systems will be able to do:

The implementation of ATOD and ATOD-Eval will fundamentally shift existing Task-Oriented Dialogue (TOD) systems from sequential, single-goal completion agents to sophisticated, agentic reasoning systems capable of managing complex, long-horizon workflows.

Here are the specific improvements and capabilities:

1. A new evaluation framework (ATOD-Eval) will be established that moves beyond simple fluency and task success rates to systematically measure advanced agentic behaviors:

2. A robust synthetic dataset (ATOD) will be created, explicitly encoding multi-goal concurrency, interleaved workflows, explicit dependencies, long-horizon memory requirements, asynchronous execution states (PENDING vs. COMPLETED), and proactive intervention opportunities.

3. An Agentic Memory System will be integrated into the core dialogue agent architecture using a dual memory store (symbolic metadata and semantic vector store) and a turn-level processing pipeline that dynamically extracts goals, checks against memory, updates state trajectories, and performs proactive auditing.

Specific Capabilities of the Improved AI System:

1. Maneuver Complex Goal Interleaving: The system will no longer process goals sequentially. It can simultaneously manage multiple objectives (e.g., booking a flight while concurrently arranging a hotel) and dynamically pause one goal (asynchronous execution, e.g., waiting for an external API response) to work on another, resuming seamlessly when the prerequisite is met.

2. Maintain Long-Horizon Context and State: The system will possess consistent state tracking across dialogues spanning many turns or even multiple sessions. It will accurately track the lifecycle of every goal—from OPEN (mentioned) to PENDING (active processing), COMPLETED, FAILED, or ABANDONED—ensuring that progress on one task does not corrupt the status of others.

3. Execute Dependency-Aware Workflows: The system will understand and manage prerequisites between goals (e.g., Payment cannot be initiated until Booking is completed). It will proactively identify blocked goals and prioritize actions necessary to satisfy dependencies, ensuring logical consistency in complex, multi-step tasks.

4. Demonstrate Proactive Assistance: The system will move beyond reactive responses by initiating helpful actions or reminding the user of pending tasks without being prompted (e.g., I see your flight is confirmed; would you like me to set a reminder to pack your passport for Sunday night?). This demonstrates true agentic initiative.

5. Achieve High Fidelity in Evaluation: By using ATOD-Eval, developers will gain fine-grained metrics (like Dependency-Aware Goal Completion Rate and Memory Recall Accuracy) that accurately reflect the performance of these advanced features, allowing for precise comparison between different LLM and memory architectures.

Agentic task-oriented dialogue (TOD) requires systems to track concurrent goals, dependencies, and long-horizon state. We examine goal-lifecycle recovery from fixed dialogue trajectories. ATOD contains 1,000 synthetic dialogues annotated for six advanced-TOD properties, and ATOD-Eval defines metrics for dependency-sensitive completion, memory recall, and proactivity. We implement a symbolic-vector memory evaluator for ATOD-Eval. Among five prompt-only predictors and six matched-backbone memory baselines, our evaluator is the only configuration above 90% in both goal detection F1 and conditional status accuracy on medium dialogues. On complex dialogues, no baseline exceeds it on both metrics; it has the highest conditional status accuracy within the memory-based block and the lowest measured per-turn latency. These experiments assess lifecycle tracking rather than interactive agent task success. A cross-family judge swap and a manual audit with 94.0% agreement provide initial checks on measurement reliability. Code and data will be released at https://github.com/amazon-science/ATOD.