Human or Machine? A Preliminary Turing Test for Speech-to-Speech Interaction
cs.AI, cs.SD
Submitted: 2026-02-27
Updated: 2026-09-08
Comments: Accepted by ICLR 2026 Conference
Code: https://github.com/Carbohydrate1001/Turing-Test
License: http://creativecommons.org/licenses/by/4.0/
The gist: The pursuit of human-like conversational agents has long been guided by the Turing test.
Terminology
Abstract
The pursuit of human-like conversational agents has long been guided by the Turing test. For modern speech-to-speech (S2S) systems, a critical yet unanswered question is whether they can converse like humans. To tackle this, we conduct the first Turing test for S2S systems, collecting 2,968 human judgments on dialogues between 9 state-of-the-art S2S systems and 28 human participants. Our results deliver a clear finding: no existing evaluated S2S system passes the test, revealing a significant gap in human-likeness. To diagnose this failure, we develop a fine-grained taxonomy of 18 human-likeness dimensions and crowd-annotate our collected dialogues accordingly. Our analysis shows that the bottleneck is not semantic understanding but stems from paralinguistic features, emotional expressivity, and conversational persona. Furthermore, we find that off-the-shelf AI models perform unreliably as Turing test judges. In response, we propose an interpretable model that leverages the fine-grained human-likeness ratings and delivers accurate and transparent human-vs-machine discrimination, offering a powerful tool for automatic human-likeness evaluation. Our work establishes the first human-likeness evaluation for S2S systems and moves beyond binary outcomes to enable detailed diagnostic insights, paving the way for human-like improvements in conversational AI systems.
Sources
- Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs
- VoiceBench: Benchmarking LLM-Based Voice Assistants
- Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
- MTalk-Bench: Evaluating Speech-to-Speech Models in Multi-Turn Dialogues via Arena-style and Rubrics Protocols
- CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models
- SycEval: Evaluating LLM Sycophancy
- An Analysis of Dialogue Repair in Virtual Voice Assistants
- OSUM-EChat: Enhancing End-to-End Empathetic Spoken Chatbot via Understanding-Driven Spoken Dialogue
- ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools
- GPT-4o System Card
- S2S-Arena: Evaluating Paralinguistic Instruction Following in Speech-to-Speech Models
- Large Language Models Pass the Turing Test
- MMAU-Pro: A Challenging and Comprehensive Benchmark for Holistic Evaluation of Audio General Intelligence
- Baichuan-Audio: A Unified Framework for End-to-End Speech Interaction
- Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities
- SeaLLMs -- Large Language Models for Southeast Asia
- Prosodically Enhanced Foreign Accent Simulation by Discrete Token-based Resynthesis Only with Native Speech Corpora
- GPT-4 Technical Report
- GPT-4 is judged more human than humans in displaced and inverted Turing tests
- MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection