Reducing Hallucination in Multimodal Large Language Models through Hard Grounding Preference Supervision

arXiv:2605.16411 · cs.CV, cs.AI, cs.CL, cs.DB, cs.LG · Submitted 2026-05-13 · Read on arXiv

cs.CV, cs.AI, cs.CL, cs.DB, cs.LG

Submitted: 2026-05-13

Updated: 2026-09-26

Terminology

• Hallucination — 74× in this paper · explained in 3 episodes
• Grounding — 44× in this paper · explained in 3 episodes
• Benchmark — 19× in this paper · explained in 2 episodes
• Visual Grounding — 12× in this paper · explained in 2 episodes
• Consistency — 6× in this paper · explained in 3 episodes
• Failure Modes — 5× in this paper · explained in 3 episodes
• Hallucination Mitigation — 5× in this paper · explained in 2 episodes
• Multimodal Reasoning — 5× in this paper · explained in 5 episodes
• Robustness — 5× in this paper · explained in 5 episodes
• Alignment — 4× in this paper · explained in 1 episode
• Direct Preference Optimization — 4× in this paper
• Fine-Tuning — 4× in this paper · explained in 5 episodes
• LLM judge — 4× in this paper · explained in 2 episodes
• Direct Preference Optimization (DPO) — 3× in this paper · explained in 9 episodes
• LLM-as-a-Judge — 3× in this paper · explained in 3 episodes
• Scaffolding — 3× in this paper · explained in 3 episodes
• Chain-of-Thought Prompting — 2× in this paper · explained in 2 episodes
• Self-consistency — 2× in this paper · explained in 2 episodes
• Vision-Language Models (VLMs) — 2× in this paper · explained in 6 episodes
• Accountability — 1× in this paper · explained in 2 episodes
• Accuracy — 1× in this paper · explained in 1 episode
• Adversarial Examples — 1× in this paper · explained in 2 episodes
• Contextual Understanding — 1× in this paper · explained in 2 episodes
• Core — 1× in this paper · explained in 2 episodes
• Deep Reinforcement Learning — 1× in this paper · explained in 2 episodes
• Distribution Shift — 1× in this paper · explained in 5 episodes
• Few-Shot Learning — 1× in this paper · explained in 4 episodes
• GRPO — 1× in this paper · explained in 6 episodes
• Iterative Refinement — 1× in this paper · explained in 3 episodes
• Multimodal Dataset — 1× in this paper · explained in 2 episodes
• RAG — 1× in this paper · explained in 3 episodes
• Reinforcement Learning — 1× in this paper · explained in 4 episodes
• Semantic Understanding — 1× in this paper · explained in 2 episodes
• Supervised Fine-Tuning (SFT) — 1× in this paper · explained in 8 episodes
• Transformer — 1× in this paper · explained in 2 episodes
• Verification — 1× in this paper · explained in 2 episodes

Sources

• MMBench: Is Your Multi-modal Model an All-around Player?
• Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigation
• Multilingual OCR-Aware Fine-Tuning and Prompt-Guided Chain-of-Thought Reasoning for Multimodal Large Language Models
• Robust Checkpoint Selection for Multimodal LLMs via Agentic Evaluation and Stability-Aware Ranking

Related papers

• Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
• AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
• Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
• MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
• TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
• A Survey on Efficient Vision-Language-Action Models