VoxPrivacy: A Benchmark for Evaluating Interactional Privacy of Speech Language Models

arXiv:2601.19956 · eess.AS, cs.AI, cs.SD · Submitted 2026-01-27 · Read on arXiv

eess.AS, cs.AI, cs.SD

Submitted: 2026-01-27

Updated: 2026-09-03

Project page: https://interactionalprivacy.github.io

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Benchmark — 54× in this paper · explained in 2 episodes
• Accuracy — 40× in this paper · explained in 1 episode
• Core — 19× in this paper · explained in 2 episodes
• Verification — 13× in this paper · explained in 2 episodes
• Large Language Model — 11× in this paper
• Fine-Tuning — 10× in this paper · explained in 5 episodes
• Robustness — 8× in this paper · explained in 5 episodes
• Ablation — 6× in this paper · explained in 1 episode
• Information Leakage — 5× in this paper · explained in 2 episodes
• LLM judge — 5× in this paper · explained in 2 episodes
• Alignment — 4× in this paper · explained in 1 episode
• Benchmarking — 4× in this paper · explained in 3 episodes
• Catastrophic Forgetting — 4× in this paper · explained in 5 episodes
• Failure Modes — 3× in this paper · explained in 3 episodes
• Invalid Responses — 3× in this paper
• Adversarial Attacks — 2× in this paper · explained in 1 episode
• Annotator Agreement — 2× in this paper · explained in 1 episode
• Consistency — 2× in this paper · explained in 3 episodes
• F1 Score — 2× in this paper · explained in 3 episodes
• Inter-Annotator Agreement — 2× in this paper
• Invalid Response Rate — 2× in this paper
• Large Language Model (LLM) — 2× in this paper · explained in 3 episodes
• True Positive — 2× in this paper
• Ablation study — 1× in this paper · explained in 1 episode
• Actor — 1× in this paper · explained in 3 episodes
• AdamW Optimizer — 1× in this paper · explained in 1 episode
• Automatic Speech Recognition — 1× in this paper
• Automatic Speech Recognition (ASR) — 1× in this paper · explained in 2 episodes
• Calibration — 1× in this paper · explained in 7 episodes
• Embeddings — 1× in this paper · explained in 2 episodes
• Equal Error Rate — 1× in this paper
• False Negative — 1× in this paper
• False Positive — 1× in this paper
• Few-Shot Learning — 1× in this paper · explained in 4 episodes
• Fidelity — 1× in this paper · explained in 2 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Ontology — 1× in this paper · explained in 2 episodes
• Transformer — 1× in this paper · explained in 2 episodes
• True Negative — 1× in this paper

Sources

• GPT-4 Technical Report
• Common Voice: A Massively-Multilingual Speech Corpus
• SafeDialBench: A Fine-Grained Safety Evaluation Benchmark for Large Language Models in Multi-Turn Dialogues with Diverse Jailbreak Attacks
• Qwen2-Audio Technical Report
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• Kimi-Audio Technical Report
• AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scale
• MTalk-Bench: Evaluating Speech-to-Speech Models in Multi-Turn Dialogues via Arena-style and Rubrics Protocols
• LLaMA-Omni: Seamless Speech Interaction with Large Language Models
• Emilia: A Large-Scale, Extensive, Multilingual, and Diverse Dataset for Speech Generation
• SOVA-Bench: Benchmarking the Speech Conversation Ability for LLM-based Voice Assistant
• AudioTrust: Benchmarking the Multifaceted Trustworthiness of Audio Large Language Models
• Baichuan-Omni-1.5 Technical Report
• DeepSeek-V3 Technical Report
• Voxtral
• Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructions
• Trust No Bot: Discovering Personal Disclosures in Human-LLM Conversations in the Wild
• Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory
• MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations
• DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressors

Related papers

• X-VC: Zero-shot Streaming Voice Conversion in Codec Space
• Autoregressive Guidance of Deep Spatially Selective Filters using Bayesian Tracking for Efficient Extraction of Moving Speakers
• Anonymization, Not Elimination: Utility-Preserved Speech Anonymization
• Towards Audio Token Compression in Large Audio Language Models
• WaveScat: Wavelet Scattering Front-Ends with Self-Supervised Features for Speech Deepfake Detection
• ProPS: Prompted Profile Synthesis for Natural Language-Conditioned Speaker Embedding Distributions