MLLMs Hallucinate when Information Distribution Drifts in Synergy Heads
cs.CV, cs.CL, cs.LG
Submitted: 2026-09-05
Updated: 2026-09-27
License: http://creativecommons.org/licenses/by/4.0/
The gist: Multimodal Large Language Models (MLLMs) often struggle with hallucinations, thus hindering their reliable practical applications.
Terminology
Abstract
Multimodal Large Language Models (MLLMs) often struggle with hallucinations, thus hindering their reliable practical applications. Existing attention-based mitigation methods mainly rely on indirect signals (e.g., attention weights) that fail to accurately reflect the actual information shift underlying hallucination generation. In this paper, we propose HEAL, Head-lEvel information disentAnglement and caLibration for identifying and mitigating hallucinations. HEAL first employs causal noise intervention on multi-head outputs to filter out causally redundant heads. Subsequently, it disentangles information distribution within the remaining heads via the counterfactual Difference-in-Differences, categorizing heads into four types. Through analysis, we observe: hallucinations happen when information distribution drifts away from a healthy equilibrium in synergy heads, not strongly correlated with the quantity or strength of modality-specific heads. Motivated by this insight, HEAL injects dynamic information calibration factors into the value vectors of synergy heads, and actively regulates visual-language dependencies, steering the output distribution towards factual evidence. Extensive experiments demonstrate that HEAL effectively reduces hallucinations across multiple MLLMs, offering a simple and interpretable pathway to enhance model trustworthiness.
Sources
- Qwen3-VL Technical Report
- Why Language Models Hallucinate
- Hallucination of Multimodal Large Language Models: A Survey
- PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Training
- Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization
- Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
- When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understanding
- Seeing Clearly by Layer Two: Enhancing Attention Heads to Alleviate Hallucination in LVLMs
- Efficient Streaming Language Models with Attention Sinks
- Nonnegative Decomposition of Multivariate Information
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
- Qwen2.5-VL Technical Report
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
- MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
- DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models
- HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding
- Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models