ImageEval 2026: Culturally Grounded Arabic Multimodal Evaluation
cs.CL, cs.AI
Submitted: 2026-08-31
Updated: 2026-08-31
Comments: Arabic LLMs, Multilingual, Multimodal, Shared Task
Project page: https://imageeval2026.github.io
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: We present an overview of the ImageEval 2026 shared task on culturally grounded Arabic multimodal evaluation.
Terminology
Abstract
We present an overview of the ImageEval 2026 shared task on culturally grounded Arabic multimodal evaluation. It includes two tasks: (i) AynVQA, covering spoken visual question answering and image-grounded hallucination detection in English and Modern Standard Arabic (MSA), and (ii) CRAI-Bench, evaluating the cultural accuracy of text-to-image generation. A total of 14 teams participated in the test phase, with 12 teams submitting system description papers. Participating systems used a range of approaches, including zero-shot prompting, fine-tuning of vision-language models, speech-recognition pipelines, ensembling, and score calibration. We describe the task setup, datasets, evaluation procedure, and participating systems, and summarize the main results across the different tracks. All datasets and evaluation scripts from the shared task are released to the research community. The shared task highlights the challenges of culturally grounded multimodal evaluation, particularly for Arabic speech and image-text reasoning.
Sources
- OASIS: A Multilingual and Multimodal Dataset for Culturally Grounded Spoken Visual QA
- LongHalQA: Long-Context Hallucination Evaluation for MultiModal Large Language Models
- AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation
- HallE-Control: Controlling Object Hallucination in Large Multimodal Models
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering