Said Aloud, Read Different: Cross-Modal Instability in Multimodal Models

arXiv:2608.27135 · cs.CL · Submitted 2026-08-27 · Read on arXiv

cs.CL

Submitted: 2026-08-27

Updated: 2026-08-27

Comments: Interpeech 2026

Code: https://github.com/baselmousi/cfhr-ci

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

The gist: Multimodal foundation models are increasingly used in speech-first assistants that must interpret spoken queries and produce visually grounded decisions.

Terminology

Abstract

Multimodal foundation models are increasingly used in speech-first assistants that must interpret spoken queries and produce visually grounded decisions. Yet it remains unclear whether semantically equivalent queries yield consistent judgments across modality (text vs. speech) and language (English vs. Arabic). We introduce a speech-augmented visually grounded contrastive triplet benchmark spanning 10,150 culturally grounded images from 18 MENA countries, where each image is paired with one supported statement and two plausible but unsupported alternatives. We define contrastive instability as the conditional rate at which a model fails to resolve all statements within a triplet, isolating fragmented reasoning from complete failure. Evaluating recent multimodal models under text and speech in English and Arabic, we find that modality and language shifts introduce substantial triplet-level inconsistencies that are not fully captured by aggregate accuracy, with speech amplifying partial failures. We make the benchmark publicly available to the community.

Sources

Related papers