Beyond Oracle Communication: Benchmarking Interactive Intent Alignment Under Miscommunication and Evolving User Intent

arXiv:2609.38604 · cs.CL, cs.AI, cs.LG · Submitted 2026-09-29 · Read on arXiv

cs.CL, cs.AI, cs.LG

Submitted: 2026-09-29

Updated: 2026-09-29

Terminology

• Benchmark — 133× in this paper · explained in 2 episodes
• Alignment — 46× in this paper · explained in 1 episode
• Verification — 29× in this paper · explained in 2 episodes
• Grounding — 23× in this paper · explained in 3 episodes
• Verifier — 21× in this paper · explained in 2 episodes
• Adaptation — 20× in this paper · explained in 1 episode
• Consistency — 12× in this paper · explained in 3 episodes
• Fidelity — 10× in this paper · explained in 2 episodes
• Robustness — 10× in this paper · explained in 5 episodes
• Benchmarking — 9× in this paper · explained in 3 episodes
• Ablation — 7× in this paper · explained in 1 episode
• Reinforcement Learning — 7× in this paper · explained in 4 episodes
• Abstraction — 6× in this paper · explained in 1 episode
• LLM judge — 6× in this paper · explained in 2 episodes
• Accuracy — 5× in this paper · explained in 1 episode
• In International Conference on Learning Representations — 5× in this paper
• Trace — 5× in this paper · explained in 2 episodes
• Generative AI — 4× in this paper · explained in 2 episodes
• Graph Construction — 4× in this paper · explained in 3 episodes
• Failure Modes — 3× in this paper · explained in 3 episodes
• Personalization — 3× in this paper · explained in 2 episodes
• Core — 2× in this paper · explained in 2 episodes
• Regret — 2× in this paper · explained in 3 episodes
• AI Agents — 1× in this paper · explained in 2 episodes
• Action Space — 1× in this paper · explained in 1 episode
• Adam — 1× in this paper · explained in 2 episodes
• Adaptive Planning — 1× in this paper · explained in 1 episode
• Commentary — 1× in this paper · explained in 2 episodes
• Fine-Tuning — 1× in this paper · explained in 5 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Human-in-the-Loop — 1× in this paper · explained in 3 episodes
• Interpretability — 1× in this paper · explained in 15 episodes
• Iterative Refinement — 1× in this paper · explained in 3 episodes
• Pragmatics — 1× in this paper · explained in 2 episodes
• Procedural Knowledge — 1× in this paper · explained in 2 episodes
• Provenance — 1× in this paper · explained in 3 episodes
• Steering — 1× in this paper · explained in 2 episodes
• Text-to-SQL — 1× in this paper · explained in 3 episodes
• World Model — 1× in this paper · explained in 7 episodes

Sources

• Drift-Bench: Diagnosing Cooperative Breakdowns in LLM Agents under Input Faults via Multi-Turn Interaction
• Trip+: Benchmarking Agents in Personalized Interactive Travel Planning
• ChatShop: Interactive Information Seeking with Language Agents
• Calibrate-Then-Act: Cost-Aware Exploration in LLM Agents
• EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent
• ClarQ-LLM: A Benchmark for Models Clarifying and Requesting Information in Task-Oriented Dialog
• From Self-Evolving Synthetic Data to Verifiable-Reward RL: Post-Training Multi-turn Interactive Tool-Using Agents
• Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agents
• BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactions
• Coding Agents Are Guessing: Measuring Action-Boundary Violations in Underspecified DevOps Instructions
• InfoPO: Information-Driven Policy Optimization for User-Centric Agents
• ClarEval: A Benchmark for Evaluating Clarification Skills of Code Agents under Ambiguous Instructions
• HorizonBench: Long-Horizon Personalization with Evolving Preferences
• EchoChain: A Full-Duplex Benchmark for State-Update Reasoning Under Interruptions
• Flipping the Dialogue: Training and Evaluating User Language Models
• Flex-TravelPlanner: A Benchmark for Flexible Planning with Language Agents
• LHAW: Controllable Underspecification for Long-Horizon Tasks
• UserRL: Training Interactive User-Centric Agent via Reinforcement Learning
• SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions
• TRIP-Bench: A Benchmark for Long-Horizon Interactive Agents in Real-World Scenarios

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering