Hallucination in Multimodal Foundation Models: A Survey on Causes, Corrections, and Evaluations

arXiv:2410.15359 · cs.AI · Submitted 2024-10-20 · Read on arXiv

cs.AI

Submitted: 2024-10-20

Updated: 2026-09-13

Code: https://github.com/lyq312318224/MLLMsAugmented

Project page: https://fuxiaoliu.github.io/LRV/https://github.com/Yuqifan1117/HalluciDoctor

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Hallucination — 160× in this paper · explained in 4 episodes
• Benchmark — 38× in this paper · explained in 2 episodes
• Accuracy — 8× in this paper · explained in 1 episode
• Instruction Tuning — 8× in this paper · explained in 2 episodes
• Core — 7× in this paper · explained in 2 episodes
• Consistency — 6× in this paper · explained in 3 episodes
• Transformer — 6× in this paper · explained in 2 episodes
• Aggregation — 5× in this paper · explained in 1 episode
• Contrastive Learning — 5× in this paper · explained in 5 episodes
• EMMA — 4× in this paper · explained in 2 episodes
• Vision-language models — 4× in this paper · explained in 2 episodes
• Diffusion Model — 3× in this paper · explained in 8 episodes
• Embeddings — 3× in this paper · explained in 2 episodes
• F1 Score — 3× in this paper · explained in 3 episodes
• Fine-Tuning — 3× in this paper · explained in 5 episodes
• Foundation Model — 3× in this paper · explained in 3 episodes
• Foundation Models — 3× in this paper · explained in 7 episodes
• Loss Function — 3× in this paper · explained in 2 episodes
• Perplexity — 3× in this paper · explained in 3 episodes
• Reinforcement Learning — 3× in this paper · explained in 4 episodes
• Diffusion Models — 2× in this paper · explained in 6 episodes
• Exposure Bias — 2× in this paper · explained in 2 episodes
• Hallucination Mitigation — 2× in this paper · explained in 2 episodes
• Modality Gap — 2× in this paper · explained in 4 episodes
• Robustness — 2× in this paper · explained in 5 episodes
• Transferability — 2× in this paper · explained in 4 episodes
• Adaptability — 1× in this paper · explained in 1 episode
• Alignment — 1× in this paper · explained in 1 episode
• Cosine Similarity — 1× in this paper · explained in 3 episodes
• Distillation — 1× in this paper · explained in 2 episodes
• Feature Extraction — 1× in this paper · explained in 2 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Generative Model — 1× in this paper · explained in 2 episodes
• Generative Models — 1× in this paper · explained in 2 episodes
• Grounding — 1× in this paper · explained in 3 episodes
• Hallucination Detection — 1× in this paper · explained in 2 episodes
• Interpretability — 1× in this paper · explained in 15 episodes
• LLM-as-a-Judge — 1× in this paper · explained in 3 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Latent Diffusion Model — 1× in this paper · explained in 2 episodes
• Latent Diffusion Models — 1× in this paper · explained in 2 episodes
• Meta-learning — 1× in this paper · explained in 2 episodes
• Prompt Engineering — 1× in this paper · explained in 4 episodes
• Pruning — 1× in this paper · explained in 4 episodes
• Representation Learning — 1× in this paper · explained in 2 episodes
• Reward Hacking — 1× in this paper · explained in 8 episodes
• Semantic Drift — 1× in this paper · explained in 3 episodes
• Verification — 1× in this paper · explained in 2 episodes
• World Model — 1× in this paper · explained in 7 episodes

Sources

• GPT-4 Technical Report
• LLaMA: Open and Efficient Foundation Language Models
• OphGLM: Training an Ophthalmology Large Language-and-Vision Assistant based on Instructions and Dialogue
• ADriver-I: A General World Model for Autonomous Driving
• Improved Baselines with Visual Instruction Tuning
• MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning
• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
• Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
• Hallucination of Multimodal Large Language Models: A Survey
• A Survey on Hallucination in Large Vision-Language Models
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• How to Bridge the Gap between Modalities: Survey on Multimodal Large Language Model
• OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models
• Scaling Instruction-Finetuned Language Models
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• MLLMs-Augmented Visual-Language Representation Learning
• MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
• CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuning
• FACTUAL: A Benchmark for Faithful and Consistent Textual Scene Graph Parsing
• Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
• VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection