DuplexWorld: Can voice agents help you get through the day?

DUPLEX WORLD introduces a benchmark for holistically evaluating speech-to-speech (S2S) voice agents across six worlds—banking, insurance, travel, healthcare, logistics, and Pathfinding—spanning 156 authored scenarios and eleven conversation types, with 3,825 scored conversations totaling 387 hours of simulated speech. The benchmark evaluates agents on three pillars: conversational dynamics (turn-taking, conversation progression, selectivity), agentic capability (goal-state check, Pass@1, Pass@3, Pass3, effort ratios), and naturalness (faithfulness, DNSMOS, UTMOS, NISQA). Five commercial systems were evaluated: Nova 2 Sonic, Gemini-3.1-Flash-Live, GPT-Realtime-2.1, GPT-Realtime-2.1-mini, and Grok Voice Think Fast 1.0.

Key results show that even the best systems leave substantial room for improvement on all three axes: Pass@1: 0.490, turn-taking: 0.653, DNSMOS: 3.378 pooled over all six worlds. Grok Voice Think Fast 1.0 leads the agentic pillar with Pass@1 of 0.490 and goal-state check of 0.779, followed by GPT-Realtime-2.1 (0.433), Gemini-3.1-Flash-Live (0.398), GPT-Realtime-2.1-mini (0.188), and Nova 2 Sonic (0.011). Reliability decays rapidly: Pass3 is roughly half of Pass@1 for the two leaders and under a third for GPT-Realtime-2.1-mini.

The paper finds that the three capabilities do not travel together. Conversational dynamics does not reproduce the agentic ordering: "GPT-Realtime-2.1 holds the best turn-taking (0.653 pooled), Grok Voice Think Fast 1.0 the best conversation progression (1.814), and Gemini-3.1-Flash-Live, third on the reward, is last on turn-taking (0.388) because it pauses to call tools. Nova 2 Sonic shows the sharpest dissociation: respectable turn-taking (0.566) and the best selectivity in every world (0.960–0.995) sit beside an agentic column of near-zeros. Naturalness separates least, with all systems within a quarter point on DNSMOS (3.13–3.40), and the two MOS leaders, GPT-Realtime-2.1 and GPT-Realtime-2.1-mini, differ by 2.3× on the reward."

A score depends on the world as much as the system: GPT-Realtime-2.1 alone spans 0.200 in Banking to 0.674 in Travel, worlds that differ only in subject matter, while the ordering barely moves. Failure-mode analysis shows engaged systems over-work in enterprise worlds, exceeding ideal workload in 50–83% of conversations with over-effort ratios of 1.7–3.0, while the disengaged system under-works in 58–95% of conversations. Pathfinding inverts this pattern: strong systems over-work while weak systems under-work.

Credential word error rate is a first-order bottleneck: GPT-Realtime-2.1 hears best: 10.4% error on spoken credentials over the telephone channel, against 14.4 for Grok Voice Think Fast 1.0, 15.5 for Gemini-3.1-Flash-Live and 20.2 for GPT-Realtime-2.1-mini, with pooled error of 17.1%. In Pathfinding, read through an explore–exploit lens, the systems that explore most arrive least: Gemini-3.1-Flash-Live and Grok Voice Think Fast 1.0 explore on 16.2% and 22.7% of moves versus 40.3–51.2% for the others, and the two lowest explorers are the two most likely to arrive, the highest explorer never arrives at all.

The paper concludes with four concrete findings: scores depend on world and conversation type as much as system; conversational and agentic capability are separate; acoustic quality tells you almost nothing about competence; and in Pathfinding, exploration correlates negatively with arrival. The practical takeaway is to report the under-effort share and the harness configuration beside every score, because both move the numbers.

Improvements to AI systems:

1. Add a credential-focused audio encoder fine-tuning stage. Train the speech encoder on noisy telephone-channel audio with synthetic misheard credentials (alphanumeric strings, dates, IDs), using contrastive loss against ground-truth transcripts. This reduces credential WER from 17.1% pooled toward <5%, directly lifting Pass@1 across all worlds.

2. Implement a dynamic effort controller with a world-aware workload prior. Use a lightweight classifier that predicts the ideal number of turns/actions per conversation type (e.g., banking = low effort, travel = high effort). The agent then adjusts its tool-call frequency and clarification rate in real time, preventing over-work (1.7–3.0× effort ratios) and under-work (58–95% of conversations) by targeting a calibrated effort band.

3. Decouple turn-taking from tool execution via speculative interruption handling. Instead of pausing to call tools (which drops turn-taking to 0.388), the agent predicts likely user responses during tool latency and pre-generates candidate replies, allowing it to interject or continue speaking without waiting. This raises turn-taking scores toward 0.8+ without sacrificing agentic accuracy.

4. Add an explore–exploit scheduler for pathfinding and multi-step tasks. Replace greedy exploration with a bandit algorithm (e.g., UCB) that tracks per-state success rates and reduces exploration after N failed moves. This directly counters the negative correlation between exploration and arrival (highest explorer never arrives), targeting arrival rates >0.7 for all systems.

5. Build a world-conditioned reward model for goal-state checking. Train a separate verifier per world (banking, insurance, etc.) that scores whether the conversation reached the goal, using world-specific state machines. This corrects the 0.200–0.674 score spread for the same system across worlds, making Pass@1 more reliable and comparable.

6. Add a naturalness–competence disentanglement layer. Post-process generated speech with a prosody re-ranker that optimizes DNSMOS/UTMOS while preserving semantic content, but also flag low-competence outputs (low Pass@1) for re-generation even if acoustics are good. This prevents systems from sounding fluent while failing task completion.

7. Implement a reliability decay mitigator for long conversations. Track per-turn confidence and automatically trigger a re-confirmation or fallback strategy when Pass@3 drops below a threshold (e.g., after 5 turns). This reduces the Pass3-to-Pass@1 halving seen in leaders, improving consistency in enterprise scenarios.

What the improved AI system can do:

• Achieve Pass@1 >0.65 pooled (up from 0.490) by reducing credential errors and world-specific variance.

• Maintain turn-taking >0.75 even during tool calls, without sacrificing goal-state accuracy.

• Keep effort ratios within 0.9–1.1× ideal workload across all worlds, avoiding both over- and under-work.

• Arrive at correct destinations in Pathfinding >70% of the time, regardless of exploration strategy.

• Produce acoustically natural speech (DNSMOS >3.5) that is also task-competent, with no dissociation between the two.

• Sustain Pass@3 above 0.6 for conversations longer than 10 minutes, preventing reliability decay.

• Report scores with under-effort share and harness configuration, enabling fair cross-system comparison.

Speech-to-speech (S2S) voice agents are increasingly being incorporated into enterprise for customer care and as daily companions for consumers owing to the ease of the conversational modality over text. However, existing benchmarks fail to holistically evaluate voice agents along axes that really matter and are shaped as tests of agentic tool calling against a database. We believe they fail to adequately account for the diversity of conversational dialogue that mundane activities introduce and further, never test how faithfully an agent can assist on tasks that move beyond database manipulation. To tackle this DuplexWorld introduces six worlds where voice agents are especially useful: banking, insurance, travel, healthcare and logistics, and Pathfinding. Agents are evaluated on eleven different types of conversations across 156 scenarios (350+ hours of conversation), each testing conversational and analytical capability to varying degrees. Through extensive evaluation comprising agentic, conversational and speech-naturalness metrics, we show that even the best voice agents leave substantial room for improvement on all 3 axes (Pass@1: 0.490, turn-taking: 0.653, DNSMOS: 3.378). We perform extensive analysis on agentic v conversational performance, world- and conversation type-wise performance, failure modes exploring the explore v exploit lens for Pathfinding conversations and voice agent reliability over all six worlds.