A Living Benchmark for Information Retrieval from Electronic Health Records

arXiv:2609.30205 · cs.AI · Submitted 2026-09-24 · Read on arXiv

cs.AI

Submitted: 2026-09-24

Updated: 2026-09-24

Code: https://github.com/alsentzerlab/brie

Terminology

• Benchmark — 62× in this paper · explained in 2 episodes
• Hallucination — 15× in this paper · explained in 3 episodes
• Redundancy — 7× in this paper · explained in 2 episodes
• Accuracy — 6× in this paper · explained in 1 episode
• Artificial Intelligence — 6× in this paper
• Failure Modes — 5× in this paper · explained in 3 episodes
• RAG — 5× in this paper · explained in 3 episodes
• Annotator Agreement — 4× in this paper · explained in 1 episode
• Robustness — 4× in this paper · explained in 5 episodes
• Agentic Retrieval — 3× in this paper · explained in 1 episode
• Alignment — 3× in this paper · explained in 1 episode
• Consistency — 3× in this paper · explained in 3 episodes
• Foundation Model — 3× in this paper · explained in 3 episodes
• Grounding — 3× in this paper · explained in 3 episodes
• Hallucination Detection — 3× in this paper · explained in 2 episodes
• Trace — 3× in this paper · explained in 2 episodes
• Verification — 3× in this paper · explained in 2 episodes
• Agentic Approaches — 2× in this paper · explained in 1 episode
• Empirical Methods in Natural Language Processing — 2× in this paper
• Foundation Models — 2× in this paper · explained in 7 episodes
• LLM-as-a-Judge — 2× in this paper · explained in 3 episodes
• Large Language Model (LLM) — 2× in this paper · explained in 3 episodes
• Verifiability — 2× in this paper · explained in 2 episodes
• Abstraction — 1× in this paper · explained in 1 episode
• Agentic System — 1× in this paper · explained in 2 episodes
• Artificial Intelligence (AI) — 1× in this paper · explained in 2 episodes
• Benchmark for Retrieving Information in EHRs — 1× in this paper
• Benchmarking — 1× in this paper · explained in 3 episodes
• Clustering — 1× in this paper · explained in 2 episodes
• Computational Biology — 1× in this paper · explained in 2 episodes
• Core — 1× in this paper · explained in 2 episodes
• Cosine Similarity — 1× in this paper · explained in 3 episodes
• Distribution Shift — 1× in this paper · explained in 5 episodes
• Embeddings — 1× in this paper · explained in 2 episodes
• Explainability — 1× in this paper · explained in 3 episodes
• Generalizability — 1× in this paper · explained in 6 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Human-in-the-Loop — 1× in this paper · explained in 3 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Multi-hop Reasoning — 1× in this paper · explained in 3 episodes
• Pairwise Comparisons — 1× in this paper · explained in 2 episodes
• Positional Bias — 1× in this paper · explained in 2 episodes
• Provenance — 1× in this paper · explained in 3 episodes
• Retrieval Augmented Generation (RAG) — 1× in this paper · explained in 4 episodes
• Semantic Similarity — 1× in this paper · explained in 2 episodes
• Temporal Reasoning — 1× in this paper · explained in 2 episodes

Sources

• A Randomized Controlled Trial and Pilot of Scout: an LLM-Based EHR Search and Synthesis Platform
• Adoption and Use of LLMs at an Academic Medical Center
• OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries
• Automating Benchmark Design
• EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• OpenAI GPT-5 System Card
• Kimi K2: Open Agentic Intelligence
• Kimi K2.5: Visual Agentic Intelligence
• Agentic Information Retrieval
• Vital Trace: Protocol-Constrained Patient-State Reasoning for Longitudinal Clinical Trajectories
• AIPatient Arena: EHR-grounded evaluation of large language models in end-to-end clinical consultation workflows
• Building the EHR Foundation Model via Next Event Prediction
• On the Robustness of Answer Formats in Medical Reasoning Models
• R2MED: A Benchmark for Reasoning-Driven Medical Retrieval
• Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
• Agentic clinical reasoning over longitudinal myeloma records: a retrospective evaluation against expert consensus
• Configurable Clinical Information Extraction with Agentic RAG: What Works, What Breaks, and Why

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
• VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection