Evaluating Retrieval-Augmented Generation vs. Long-Context Input for Clinical Reasoning over EHRs

arXiv:2508.14817 · cs.CL, cs.AI · Submitted 2025-08-20 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Evaluating Retrieval-Augmented Generation vs. Long-Context Input for Clinical Reasoning over EHRs".

Jane: The paper was written by Skatje Myers, Dmitriy Dligach, Timothy A. Miller, Samantha Barr, James Landefeld et al. from University of Wisconsin-Madison and Loyola University Chicago and Boston Children’s Hospital and Harvard Medical School and University of Colorado-Anschutz.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title: Tom: So, "Evaluating Retrieval-Augmented Generation versus Long-Context Input" is essentially asking which method is better at performing complex clinical reasoning over patient data. It’s not just a simple question for the models; they're being tested on real-world hospital records.

Jane: Think of it as a challenge to see if AI can manage the complexity of a patient’s entire medical story without getting overwhelmed by all that extra documentation that doesn't matter at once.

Lu: The authors, by looking at this, are trying to determine if the latest advancements in models—the ones capable of handling up to 128K tokens—are truly necessary for every single task we ask them to perform.

Meng: I’m particularly interested in the practical implication for cost. If a complex reasoning task can be solved using RAG, we are processing far less data than if we feed the full context window, which directly translates to lower operational costs.

Lalam: That efficiency is crucial for scalability in healthcare. We need systems that can handle massive patient populations without requiring exponentially more computational power as the note volume grows.

Tom: It's a huge shift in how we think about AI interaction with medical records. It's moving away from simply processing everything toward finding the right information, which is a concept that has big implications for our future-focused healthcare strategy.

Jane: We’ll summarize exactly what they found when we move to Segment three showing how these methods stack up against each other.

Summary of Findings: Tom: In the summary, the authors tested three distinct clinical tasks: identifying imaging procedures, tracking antibiotic timelines, and generating key diagnoses. The overall findings are quite encouraging for RAG.

Jane: Generally speaking, RAG demonstrated a strong ability to perform these tasks efficiently across all three models they used—GPT-five point four-mini, Mistral Medium three and DeepSeek V3 point 1.

Lu: The key finding is that RAG wasn't just competitive; in several areas, it significantly outperformed the baseline of providing only the most recent clinical notes without any retrieval mechanism at all.

Meng: For tasks like identifying imaging procedures, the performance gain using RAG was substantial, sometimes reaching a two point five-fold improvement over the simple recent-note approach. That level of difference is hard to ignore in a real hospital setting.

Lalam: It shows that even when facing complex records, targeted retrieval methods are highly effective at helping achieve accurate results without needing massive context input for the whole model.

Tom: But it’s not a uniform success, although we see clear wins in some areas. We also saw that while RAG was often better, the performance on generating key diagnoses remained quite static across both methods and models.

Jane: It seems like there are limits to how much the AI can extract meaningful information from the clinical record for certain types of questions.

Tom: That leads us into Segment four where we’ll explore exactly *why* RAG succeeds in some areas and why other areas remain challenging.

Improvements: Tom: We saw that the most consistent gains from RAG were in the Imaging Procedures task, which was a relatively straightforward extraction of information data. It required the model to find specific details like modality and location.

Jane: And even though it was straightforward, RAG still delivered massive performance improvements—we're talking about reaching nearly three hundred eighty-seven percent F1 scores in some successful retrieval scenarios. That's a huge jump from the initial baseline.

Lu: The reason it works so well in imaging is that the necessary information is clearly defined and located within the targeted passages, allowing us to bypass vast amounts of irrelevant text.

Meng: Then we have the Antibiotic Timelines task, which requires more medical reasoning than just looking at a list. But here too, RAG provided significant gains over just using recent notes alone, often between twenty-two percent and thirty-three percent better on average.

Lalam: It’s interesting that for the timelines, the performance gained quickly and then plateaued, suggesting we only need a limited number of passages to accurately reconstruct a patient's history.

Tom: That’s a huge improvement over simply giving long context, which can sometimes get "lost in the middle" of massive text blocks. The gain is both substantial and consistent across models for these tasks.

Jane: However, the Diagnosis Generation task proved to be the biggest challenge for any method or model, showing very little difference in performance whether it used targeted retrieval or full long context input.

Tom: This suggests that either the documentation variability is too high, or maybe even our methods of measuring those diagnoses are hitting a ceiling that neither RAG nor long context can overcome.

Jane: We’ll wrap up the discussion and summarize what this all means for the future in Segment five so let's look at the big picture.

Conclusion: Tom: So, to wrap up our discussion of "Evaluating Retrieval-Augmented Generation vs. Long-Context Input for Clinical Reasoning over EHRs," it’s clear that a simple targeted retrieval approach is highly competitive and very efficient.

Jane: Even though newer models are becoming incredibly good at handling long sequences of text, RAG provides a powerful, focused alternative that performs similarly to using up 120K tokens in many cases.

Lu: This suggests we don't need to rely solely on ever pushing context windows further; we can achieve high performance with surgical precision instead.

Meng: For the practical application of this means that implementing RAG is a very viable, cost-effective way to deploy AI in clinical settings without incurring massive computational overhead.

Lalam: I see this as a major step toward better, more focused AI applications in healthcare that dramatically improves how we manage patient information and support critical decision-making.

Tom: It’s clear the future of targeted retrieval is bright, and it looks like our investigation into RAG versus long-context input has given us a very compelling answer for the next chapter of this paper.

Jane: We've had a great discussion today on how targeted retrieval is proving to be a highly effective and efficient approach for clinical tasks over large amounts of EHR data.

Tom: Thanks to all our guests—Lu, Meng, and Lalam—for joining us! We’ll be back next time with another exciting paper. Goodbye everyone!

University of Wisconsin-Madison · Loyola University Chicago · Boston Children’s Hospital · Harvard Medical School · University of Colorado-Anschutz

cs.CL, cs.AI

Submitted: 2025-08-20

Updated: 2026-07-09

Importance score: 82/100

The gist: The paper details rigorous methodological evaluations of advanced Natural Language Processing (NLP) techniques designed for extracting and normalizing complex medical concepts from unstructured

Key concepts

Retrieval-Augmented Generation (RAG)
RAG utilizes targeted retrieval methods to find specific, relevant information within a patient's extensive medical records (EHR). Instead of processing all data, it focuses only on necessary passages, which provides high accuracy and significantly lowers computational costs.
Long-Context Input
This method involves feeding an AI model its entire medical history by utilizing models capable of handling very large sequences, such as 128K tokens. The study tested if this massive input was necessary for every task compared to targeted retrieval.
Clinical Reasoning over EHR
This is the process of AI analyzing real-world patient data from Electronic Health Records (EHRs). It involves performing complex tasks, such as tracking medication timelines or identifying medical procedures, to assist in clinical decision-making.

Terminology

Summary

The paper details rigorous methodological evaluations of advanced Natural Language Processing (NLP) techniques designed for extracting and normalizing complex medical concepts from unstructured Electronic Health Record (EHR) text. These findings are critical because accurate diagnosis extraction underpins downstream clinical decision support, billing, and research efforts in healthcare AI. The work assesses multiple pipelines—including SNOMED CT entity linking, ICD-10 normalization, and the comparison between retrieval-augmented generation (RAG) approaches versus pure long-context modeling—to determine the most reliable methods for capturing clinically significant diagnoses.

SNOBERT Training and Vocabulary Updates

The authors detail the training of a specialized model named SNOBERT. This model was trained using a specific configuration derived from the SNOMED CT Entity Linking Challenge data. A key methodological point is the use of updated vocabulary files, specifically the International SNOMED vocabulary files from 2025, as access to the challenge's original version was unavailable. The study notes that while they trained a single model, this approach was deemed sufficient after expert review determined its performance on the downstream ICD-10 code extraction step to be acceptable.

Validation of Filtered Diagnosis Extraction

To test the reliability of filtering billing codes down to diagnoses relevant to a discharge summary, a validation task was performed. A physician independently reviewed a random sample of 20 hospitalizations, comparing their filtered list against the model's output. This manual annotation allowed for the calculation of key performance metrics:

  • Precision: 86.72

  • Recall: 92.50

  • F1 Score: 89.52

These results are interpreted as evidence that the Filtered gold standard reasonably approximates clinician judgment for this filtering task.

Free-text to ICD-10 Normalization Pipeline Assessment

The reliability of the entire text normalization pipeline—which involves SNOBERT-based SNOMED extraction followed by mapping to ICD-10—was assessed across 40 cases. The evaluation utilized a detailed scoring rubric (Score 5 being Essentially correct, and Score 1 being Largely wrong). The analysis revealed differences in performance based on the source text:

  • Discharge Summaries: The mean score was calculated at 3.65.

  • LLM Outputs: The mean score was calculated at 4.10.

The authors concluded that the normalization quality was somewhat higher on LLM outputs than on discharge summaries, attributing this difference to the fact that LLM outputs tend to use more explicit, standardized diagnostic phrasing, whereas discharge summaries contain challenging elements like abbreviations and narrative phrasing.

Improvements for AI systems

Based on a detailed review of this research methodology concerning clinical NLP, semantic normalization, and diagnostic extraction from complex EHR data, I have identified three critical areas for architectural improvement. These improvements aim to move beyond simple information retrieval and achieve true clinical reasoning within the AI system.

Here are the proposed improvements and the resulting capabilities of the enhanced AI system:

Area of Improvement: Enhancing time-series extraction from unstructured text (as demonstrated in antibiotic tracking). Current systems often treat temporal events linearly, failing to model complex causal relationships between interventions and conditions.

Specific Enhancement: The system must be upgraded from a sequence tagger to a Graph Neural Network (GNN) architecture trained specifically on the causality and timing of medical events. This graph must link:

  1. The Condition Node: (e.g., Pneumonia, Septic Shock).

  2. The Intervention Node: (e.g., Ceftriaxone administration).

  3. The Temporal Edge: Which must not only hold a start/end date but also a Reason For weight.

Improved AI Capability: The system can now perform Evidence-Based Treatment Mapping. Instead of simply listing when an antibiotic was given, it will generate a verifiable graph showing: "Antibiotic X was administered between Date A and Date B because of Diagnosis Y, which peaked on Date C." This capability drastically reduces the risk of misinterpreting routine continuation doses versus necessary acute treatment.

Area of Improvement: Improving the reliability and clinical rigor during the free-text to SNOMED to ICD-10 pipeline, especially when bridging highly specific narrative text to generalized billing codes (as discussed in Sections 7.4 and 7.5).

Area of Improvement: Overhauling the evaluation metric (Table A5) by replacing simple percentage-based scoring with a risk-weighted system that quantifies the clinical impact of an omission or misclassification.

The final error score must be calculated as the sum of the weights of all missed or incorrectly coded diagnoses (sum W s).

Sources

Related papers