Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluation

arXiv:2606.17188 · cs.CV, cs.CL · Submitted 2026-06-15 · Read on arXiv

cs.CV, cs.CL

Submitted: 2026-06-15

Updated: 2026-08-30

Code: https://github.com/prabhjotschugh/Not-Truly-Multilingual-PuMVR

Terminology

• Accuracy — 29× in this paper · explained in 1 episode
• Benchmark — 25× in this paper · explained in 2 episodes
• Grounding — 11× in this paper · explained in 3 episodes
• Consistency — 10× in this paper · explained in 3 episodes
• Script Consistency Rate — 6× in this paper
• Visual Grounding — 5× in this paper · explained in 2 episodes
• Few-Shot Lift — 4× in this paper
• Multimodal Reasoning — 4× in this paper · explained in 5 episodes
• Robustness — 4× in this paper · explained in 5 episodes
• Ablation — 2× in this paper · explained in 1 episode
• Benchmarking — 2× in this paper · explained in 3 episodes
• In-Context Learning — 2× in this paper · explained in 4 episodes
• Prevalence-Adjusted Bias-Adjusted Kappa — 2× in this paper
• Tokenization — 2× in this paper · explained in 2 episodes
• Vision-Language Models (VLMs) — 2× in this paper · explained in 6 episodes
• Visual Gain — 2× in this paper
• Adaptation — 1× in this paper · explained in 1 episode
• Alignment — 1× in this paper · explained in 1 episode
• Annotator Agreement — 1× in this paper · explained in 1 episode
• Entanglement — 1× in this paper · explained in 3 episodes
• Evaluation Dimensions — 1× in this paper · explained in 3 episodes
• Explainability — 1× in this paper · explained in 3 episodes
• F1 Score — 1× in this paper · explained in 3 episodes
• Few-Shot Learning — 1× in this paper · explained in 4 episodes
• Generalizability — 1× in this paper · explained in 6 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Generative AI — 1× in this paper · explained in 2 episodes
• Instruction Tuning — 1× in this paper · explained in 2 episodes
• Transfer Learning — 1× in this paper · explained in 10 episodes
• Transferability — 1× in this paper · explained in 4 episodes

Sources

• Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
• IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages
• MaXM: Towards Multilingual Visual Question Answering
• IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs
• Script Gap: Evaluating LLM Triage on Indian Languages in Native vs Romanized Scripts in a Real World Setting
• PALO: A Polyglot Large Multimodal Model for 5B People
• Learning Transferable Visual Models From Natural Language Supervision
• MVL-SIB: A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matching
• BLEnD-Vis: Benchmarking Multimodal Cultural Understanding in Vision Language Models
• All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages

Related papers

• Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
• AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
• Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
• MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
• TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
• A Survey on Efficient Vision-Language-Action Models