PARSA-Bench: A Comprehensive Persian Audio-Language Model Benchmark
cs.CL, cs.SD
Submitted: 2026-03-15
Updated: 2026-09-14
License: http://creativecommons.org/licenses/by/4.0/
The gist: Persian poses unique audio understanding challenges through its classical poetry, traditional music, and pervasive code-switching, none of which is captured by existing benchmarks.
Terminology
Abstract
Persian poses unique audio understanding challenges through its classical poetry, traditional music, and pervasive code-switching, none of which is captured by existing benchmarks. We introduce PARSA-Bench (Persian Audio Reasoning and Speech Assessment Benchmark), the first dedicated benchmark for evaluating LALMs on Persian language and culture. It covers 16 tasks, ten of them new, spanning speech understanding, paralinguistic analysis, and culturally grounded audio reasoning. Across most tasks, text-only baselines outperform their audio counterparts, so audio understanding rather than language knowledge remains the main limitation, and supplying the transcript alongside the audio lifts weak models to near their text-only level. The consistent exception is Persian poetry, where prosody carries information the written form cannot: audio beats text on both poetry tasks, and metre detection shows the first signs of being learnable only at the largest model scale. The dataset is publicly available at: https://huggingface.co/datasets/MohammadJRanjbar/PARSA-Bench
Sources
- GPT-4 Technical Report
- LLaMA: Open and Efficient Foundation Language Models
- Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models
- SALMONN: Towards Generic Hearing Abilities for Large Language Models
- Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities
- SoundMind: RL-Incentivized Logic Reasoning for Audio-Language Models
- MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark
- MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix
- MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark
- Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks
- MMAU-Pro: A Challenging and Comprehensive Benchmark for Holistic Evaluation of Audio General Intelligence
- TinyStories: How Small Can Language Models Be and Still Speak Coherent English?
- ShEMO -- A Large-Scale Validated Database for Persian Speech Emotion Detection
- Persian Musical Instruments Classification Using Polyphonic Data Augmentation
- Qwen2.5-Omni Technical Report
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering