When Words Speak Louder than Images: Towards Understanding Language Bias in Vision-Language Models

arXiv:2609.35272 · cs.CL · Submitted 2026-09-28 · Read on arXiv

cs.CL

Submitted: 2026-09-28

Updated: 2026-09-28

Terminology

• Association for Computational Linguistics — 24× in this paper
• Hallucination — 22× in this paper · explained in 4 episodes
• Vision-language models — 22× in this paper · explained in 2 episodes
• Grounding — 14× in this paper · explained in 3 episodes
• Large Language Model — 6× in this paper
• Conference on Computer Vision and Pattern Recognition — 5× in this paper
• Trace — 5× in this paper · explained in 2 episodes
• Accuracy — 4× in this paper · explained in 1 episode
• Generative AI — 4× in this paper · explained in 2 episodes
• Benchmark — 3× in this paper · explained in 2 episodes
• Generalization — 3× in this paper · explained in 13 episodes
• Interpretability — 3× in this paper · explained in 15 episodes
• Hallucination Mitigation — 2× in this paper · explained in 2 episodes
• Multimodal Inputs — 2× in this paper · explained in 2 episodes
• Activation Patching — 1× in this paper · explained in 2 episodes
• Alignment — 1× in this paper · explained in 1 episode
• Attention Mechanism — 1× in this paper · explained in 3 episodes
• CVF International Conference on Computer Vision — 1× in this paper
• Compositional Generalization — 1× in this paper · explained in 4 episodes
• Consistency — 1× in this paper · explained in 3 episodes
• Distillation — 1× in this paper · explained in 2 episodes
• Distribution Matching — 1× in this paper · explained in 2 episodes
• Downstream Tasks — 1× in this paper · explained in 2 episodes
• Failure Modes — 1× in this paper · explained in 3 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Foundation Models — 1× in this paper · explained in 7 episodes
• Large Language Model (LLM) — 1× in this paper · explained in 3 episodes
• Latent Space — 1× in this paper · explained in 7 episodes
• Mechanistic interpretability — 1× in this paper · explained in 2 episodes
• Model-Agnostic — 1× in this paper · explained in 2 episodes
• Multimodal Learning — 1× in this paper · explained in 2 episodes
• Representation Learning — 1× in this paper · explained in 2 episodes
• Semantic Alignment — 1× in this paper · explained in 3 episodes
• Steering — 1× in this paper · explained in 2 episodes
• Transformer — 1× in this paper · explained in 2 episodes
• Unified Framework — 1× in this paper · explained in 5 episodes
• Visual Grounding — 1× in this paper · explained in 2 episodes

Sources

• LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training
• Qwen2.5-VL Technical Report
• Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic
• Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating
• Gemma 3 Technical Report
• OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation
• Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs
• Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts
• Are Reasoning Vision-Language Models Robust to Semantic Visual Distractions?
• What Makes "Good" Distractors for Object Hallucination Evaluation in Large Vision-Language Models?
• Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decoding

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering