A frontend-backend architecture for tool calls in full-duplex speech models

arXiv:2609.19334 · cs.CL · Submitted 2026-09-16 · Read on arXiv

cs.CL

Submitted: 2026-09-16

Updated: 2026-09-18

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Accuracy — 5× in this paper · explained in 1 episode
• Benchmarking — 5× in this paper · explained in 4 episodes
• Benchmark — 4× in this paper · explained in 2 episodes
• LLM judge — 3× in this paper · explained in 2 episodes
• Agentic Task — 2× in this paper · explained in 1 episode
• Agentic Tasks — 2× in this paper · explained in 1 episode
• AI Agents — 1× in this paper · explained in 2 episodes
• Abstract Syntax Tree — 1× in this paper
• Agentic Reasoning — 1× in this paper · explained in 1 episode
• Berkeley Function-Calling Leaderboard — 1× in this paper
• Block Diffusion — 1× in this paper · explained in 2 episodes
• Fidelity — 1× in this paper · explained in 2 episodes
• Fine-Tuning — 1× in this paper · explained in 5 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Supervised Fine-Tuning (SFT) — 1× in this paper · explained in 8 episodes
• Transformer — 1× in this paper · explained in 2 episodes

Sources

• Moshi: a speech-text foundation model for real-time dialogue
• Covo-Audio Technical Report
• Easy Turn: Integrating Acoustic and Linguistic Modalities for Robust Turn-Taking in Full-Duplex Spoken Dialogue Systems
• $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
• Audio2Tool: Speak, Call, Act -- A Dataset for Benchmarking Speech Tool Use
• Kimi-Audio Technical Report
• Qwen3-Omni Technical Report
• $\tau$-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains
• DuplexSLA: A Full-Duplex Spoken Language Model with Synchronized Speech, Language, and Action
• KAME: Tandem Architecture for Enhancing Knowledge in Real-Time Speech-to-Speech Conversational AI
• MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Models
• EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents
• Enabling Streaming User Transcription in Full-Duplex Speech-to-Speech Models
• Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
• Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency
• Chatterbox-Flash: Prior-Calibrated Block Diffusion for Streaming Zero-Shot TTS
• VoiceBench: Benchmarking LLM-Based Voice Assistants
• Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluation

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering