MedHal: a Synthetic Dataset for Medical Hallucination Detection

arXiv:2504.08596 · cs.CL, cs.AI · Submitted 2025-04-11 · Read on arXiv

cs.CL, cs.AI

Submitted: 2025-04-11

Updated: 2026-09-25

Terminology

• Hallucination — 55× in this paper · explained in 3 episodes
• Hallucination Detection — 29× in this paper · explained in 2 episodes
• Question Answering — 19× in this paper
• Information Extraction — 17× in this paper
• Benchmark — 12× in this paper · explained in 2 episodes
• Fine-Tuning — 11× in this paper · explained in 5 episodes
• Natural Language Inference — 11× in this paper
• F1 Score — 4× in this paper · explained in 3 episodes
• Ablation — 3× in this paper · explained in 1 episode
• Accuracy — 3× in this paper · explained in 1 episode
• Natural Language Inference (NLI) — 3× in this paper · explained in 2 episodes
• Adam — 2× in this paper · explained in 2 episodes
• Benchmarking — 1× in this paper · explained in 3 episodes
• Consistency — 1× in this paper · explained in 3 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Foundation Models — 1× in this paper · explained in 7 episodes
• LLM-as-a-Judge — 1× in this paper · explained in 3 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Multi-hop Reasoning — 1× in this paper · explained in 3 episodes
• Transformer — 1× in this paper · explained in 2 episodes
• Transformer Models — 1× in this paper · explained in 2 episodes
• alpha — 1× in this paper · explained in 1 episode

Sources

• A Data-Centric Approach To Generate Faithful and High Quality Patient Summaries with Large Language Models
• MedHalu: Hallucinations in Responses to Healthcare Queries by Large Language Models
• What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams
• Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models
• Detecting and Evaluating Medical Hallucinations in Large Vision Language Models
• SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization
• MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering
• The Llama 3 Herd of Models
• Med-HALT: Medical Domain Hallucination Test for Large Language Models

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering