Text Knows What, Tables Know When: Clinical Timeline Reconstruction via Retrieval-Augmented Multimodal Alignment

arXiv:2605.15168 · cs.CL, cs.AI, cs.LG, stat.ML · Submitted 2026-05-14 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Text Knows What, Tables Know When".

Jane: Detailed Research Summary of "Text Knows What, Tables Know When:

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Let's talk about the title itself: "Text Knows What, Tables Know When." It really captures the core idea—the text tells you what happened, but the tables give you the 'when.' But it’s not just a catchy phrase; it points to a specific technical approach.

Jane: The authors are trying to solve that problem where clinical notes are rich in context but weak on exact timestamps. They argue that unstructured text gives us the narrative content, but structured data provides the temporal anchors needed for accuracy.

Lu: They focus on using retrieval augmented multimodal alignment to bridge that gap, which means they’re pulling in those structured EHR rows—the tables—to refine the timeline derived from the text, specifically to resolve that wide and overlapping uncertainty we see in purely text-based reconstructions.

Meng: So, a listener might ask, "How does this actually work if the narrative is so vague?" They're implying that by grounding the text events with structured data points—like a specific lab result timestamp—the model gets much tighter timing.

Lalam: It’s like having two different views of the same event. The text gives you the patient's experience, and the table gives you a verifiable time marker, which is what makes this multimodal alignment so powerful for fixing those timeline issues.

Tom: Exactly! And they’ve shown that this isn't just about getting a better summary; it’s about getting absolute timestamp accuracy improved across nearly all tested models when using this retrieval-augmented approach.

Jane: That’s the main promise: improving absolute timestamp accuracy without messing up how well the AI understands what the patient was actually experiencing. It keeps the content intact while tightening the timing.

Lu: The authors point out that they are modeling this as a graph-based multistep process, starting with central anchor events from text and then strategically placing other events before using those structured EHR rows for external calibration.

Meng: That decomposition is key, I think. It shows they aren't just throwing all the data at the wall; they have a specific plan for how the different types of information should interact sequentially.

The paper's summary: Tom: So, let’s get into what this paper actually does in detail. The core idea is that reconstructing a clinical timeline is hard because events in the narrative are often just loosely ordered, and that text alone can’t tell you if fatigue came before or after the speech difficulty.

Jane: They summarize the problem as a gap between what unstructured narratives offer—deep context about symptom progression—and what they lack—precise temporal markers for those symptoms. The paper introduces this retrieval-augmented multimodal alignment framework to fix that specific gap.

Lu: The methodology involves this graph-based approach: you start by extracting central anchor events from the narrative, which form the backbone of your timeline. Then, you try to place everything else relative to that backbone.

Meng: And here’s where they use those structured EHR rows—they retrieve relevant tabular data—to calibrate those placements, essentially correcting the inherent ambiguities in the text-only sequence with external temporal evidence.

Lalam: They found that this calibration step is what provides the real lift; it’s like using a map to correct a GPS when your narration keeps taking you down wrong streets. The paper shows that when structured data calibrates events, you see much better absolute timestamp accuracy compared to just using the text alone.

Tom: It seems they are very specific about where this works best. They found that introducing the structured evidence at certain stages—like first stabilizing the central timeline before refining the whole thing later—was more effective than just applying it all at once.

Jane: That’s a really practical insight for anyone trying to build a system like this, because it tells you *when* you should introduce your external calibration data for the best results.

Lu: It confirms that structured data isn't just noise; it’s a temporal calibrator that refines the trajectory derived from the narrative content. This is important because it validates using tables not as a substitute, but as an external check on the text's timing.

The paper's improvements: Tom: Now for the results, because that’s where we see if this actually works in practice. The main improvement they highlight is a consistent boost in absolute timestamp accuracy across all evaluated models compared to just using text-only reconstruction methods.

Jane: They state that when they use this retrieval-augmented multimodal pipeline, the absolute timestamp accuracy goes up across nearly every model tested on those MIMIC datasets, and interestingly, this happens without lowering the rate at which the AI correctly identifies which events actually match in the narrative.

Meng: So they didn't just get better at guessing the time; they got better at aligning time while still being good at identifying what things are. That’s a solid result for practical applications where you need both content and timing.

Lu: They also found that central events—the ones that form the backbone of the timeline—show substantial gains in ordering metrics like concordance and anchored concordance, which suggests that getting those main points right sets up a much more robust global structure.

Lalam: The paper also made a very important observation about what's missing: they found that about thirty-four point eight percent of events derived from the text simply don’t appear in the structured EHR records at all. That tells us that narrative text is essential for capturing things like symptom progression or severity quantification that get lost in the structured format.

Tom: That missing data point is huge because it proves that tables aren't a complete picture; they are just one view of the patient journey, and we still need the text to understand the full trajectory.

Jane: So, while those thirty-four point eight percent missing events show where structured data falls short on narrative content, the improved accuracy on supported events shows how much better we can localize those specific points when they are cross-referenced correctly.

Conclusion: Tom: Wrapping up this look at "Text Knows What, Tables Know When," the main thing is that narrative text and structured EHR data aren't competing sources of truth; they’re complementary tools for clinical timeline reconstruction. The authors strongly argue that the retrieval-augmented, multistep scaffolded pipeline is a very effective way to achieve clinically faithful and precise reconstructions.

Jane: They conclude that the text provides the vital content and context, while the structured EHR data acts as an indispensable external evidence source to sharpen and correct how those events are timed in a patient’s history. It's about synthesis, not replacement.

Lu: From my perspective, this paper validates treating structured data as a temporal calibrator rather than just another piece of data to be processed independently; it’s about using it strategically at different points in the pipeline to stabilize and then refine the timeline structure.

Meng: Practically speaking, this means that for building tools that need to track complex patient courses, you need a system that can handle both the narrative richness and the structured temporal constraints simultaneously. It’s a necessary combination for reliable AI applications in healthcare.

Lalam: I think what stands out most is how they showed the importance of the multistep decomposition—that specific sequence matters for where you apply your multimodal evidence to get those best absolute timestamp accuracy scores.

Tom: So, to summarize this paper, "Text Knows What, Tables Know When: Clinical Timeline Reconstruction via Retrieval-Augmented Multimodal Alignment," we’ve seen that combining narrative and tabular data through this layered approach significantly improves the precision of clinical timelines by using structured rows as external temporal anchors.

Jane: It gives us a clear picture: the text handles the 'what' and 'how,' and the tables handle the precise 'when.' It sets a high bar for how we should think about merging these two different types of information in AI systems.

Lu: We’re looking forward to seeing how future work might build on this graph-based scaffolding to handle even more complex, multi-faceted trajectories, perhaps incorporating more dynamic temporal modeling as suggested by other related research.

Meng: For the practical side, I'm interested in how we can make this calibration step work robustly when the narrative text is extremely noisy or ambiguous, which is often the case in real-world clinical notes. That’s where engineering challenges will lie next.

Lalam: It’s a solid piece of research because it doesn't just show a result; it shows *why* that result happens by breaking down the process into stages and showing exactly where the structured data adds value.

Sayantan Kumar, Shahriar Noroozizadeh, Juyong Kim, Jeremy C. Weiss

National Library of Medicine · Carnegie Mellon University

cs.CL, cs.AI, cs.LG, stat.ML

Submitted: 2026-05-14

Updated: 2026-10-05

Comments: Accepted for oral presentation at the Pacific Symposium on Biocomputing (PSB) 2027. Sayantan Kumar, Shahriar Noroozizadeh, Juyong Kim (authors contributed equally)

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 92/100

The gist: Detailed Research Summary of "Text Knows What, Tables Know When: Clinical Timeline Reconstruction via Retrieval-Augmented Multimodal Alignment" This research introduces a novel retrieval-augmented

Key concepts

Retrieval-Augmented Multimodal Alignment
This is a framework that aligns two different types of data—unstructured text and structured tables—by retrieving relevant information from the tables to help interpret the text. It uses both modalities together to create a more accurate understanding of clinical events, ensuring the timeline derived from text is grounded in factual, structured evidence.
Central Anchor Events
These are the most important events identified directly from the patient's written narrative. They form the main backbone or foundation of the entire clinical timeline reconstruction process. Once these core events are established, other less important events are placed around them to build a preliminary structure.
Temporal Calibrator
Structured EHR data acts as a calibrator for the text-based timeline. Instead of replacing the narrative content, the tables provide precise timestamps that correct and refine when specific events occurred in real life. This external evidence sharpens the temporal localization of events mentioned in the text.

Terminology

Summary

Detailed Research Summary of Text Knows What, Tables Know When: Clinical Timeline Reconstruction via Retrieval-Augmented Multimodal Alignment

This research introduces a novel retrieval-augmented multimodal alignment framework designed to bridge the critical gap between unstructured clinical narratives (text) and structured electronic health record (EHR) data. The primary objective of this work is to significantly improve the temporal precision of absolute clinical timelines extracted from narrative text by leveraging external, structured temporal evidence.

Core Methodology: A Graph-Based Multistep Process

The framework models timeline reconstruction as a sophisticated, graph-based multistep process rather than a single monolithic task. This decomposition reflects a clinically intuitive understanding of event relationships:

  1. Central Scaffold Construction: The process begins by extracting key central anchor events directly from the unstructured clinical narratives. These anchors form the foundational temporal backbone of the timeline.

  2. Relative Placement: Subsequent, non-central events are then strategically placed relative to this established backbone, establishing an initial temporal structure.

  3. External Calibration (The RAG Component): The crucial step involves calibrating this scaffold using retrieved structured EHR rows. These retrieved tabular records serve as external temporal evidence to refine the timing of the text-derived events, effectively correcting inaccuracies in the narrative timeline.

  4. Full Assembly and Refinement: Following calibration, the full timeline is assembled and subjected to final refinement stages.

The core intuition driving this approach is that structured EHR data functions optimally not as a substitute for narrative event content, but rather as a temporal calibrator for the trajectories derived from text.

Key Contributions and Empirical Findings

The paper details several significant contributions, supported by rigorous evaluation across instruction-tuned large language models (LLMs) on the i2m4 benchmark spanning MIMIC-III and MIMIC-IV datasets:

1. Improved Temporal Precision via Multimodal Integration:

The primary finding is that the retrieval-augmented multimodal pipeline consistently improves absolute timestamp accuracy (AULTC) and enhances temporal concordance across nearly all evaluated models compared to unimodal text-only reconstruction. Crucially, this improvement in temporal quality occurs without compromising event match rates.

2. Multistep Decomposition Efficacy:

The multistep decomposition is empirically validated. Stage-specific integration analysis demonstrated that structured evidence is most effective when introduced at specific points in the pipeline—for instance, calibrating the central timeline first to stabilize the backbone, and then refining the full timeline later. This confirms that structured data's utility depends on its stage of application.

3. The Role of Structured Data vs. Narrative Text:

The empirical gap analysis provides a sobering but vital insight: 34.8% of text-derived events are entirely absent from the tabular EHR records. This finding strongly motivates the necessity of unstructured text, as it preserves critical clinical information—such as symptom progression, severity quantification, and causal context—that is frequently compressed or delayed in structured formats. Therefore, narrative text remains essential for capturing the content of the patient trajectory.

4. Differentiated Performance Patterns:

The evaluation metrics reveal nuanced performance characteristics:

  • Central Events: These events yield substantial gains in ordering metrics (concordance and anchored concordance) across all models, indicating that they provide a robust global temporal structure.

  • Events Supported by Structured Evidence: These specific events consistently achieve superior absolute timestamp accuracy (AULTC) and often better temporal ordering.

  • Model Dependence: While overall patterns are consistent, the degree of concordance remains more model-dependent than absolute scores, though the multimodal variant generally shows higher AULTC across all models.

Conclusion: The Synthesis of Modalities

The overall conclusion firmly establishes a complementary relationship between the modalities: narrative text is the primary source for event content and context, while structured EHR data serves as indispensable complementary evidence to sharpen and correct the temporal localization of those events. The retrieval-augmented, multistep scaffolded pipeline is presented as a highly valuable technique for achieving clinically faithful and temporally precise patient trajectory reconstructions.

Improvements for AI systems

  1. Bold Header: Multimodal Temporal Calibration for Enhanced Precision

This improvement involves deploying a retrieval-augmented multimodal alignment framework that bridges this gap [between unstructured clinical narratives and structured EHR data] to improve the temporal precision of absolute clinical timelines extracted from text. The improved AI system can reconstruct precise patient trajectories by utilizing retrieved structured EHR rows as external temporal evidence to calibrate text-derived events, thus resolving the wide and overlapping uncertainty found in unimodal reconstructions.

  1. Bold Header: Graph-Based Multistep Reconstruction

The AI system should adopt a graph-based multistep pipeline that first identifies temporally informative central events, builds an initial scaffold, and then refines it using structured evidence. This decomposition reflects a clinically natural distinction between events that define the backbone of an encounter and events whose timing is better interpreted relative to that backbone.

  1. Bold Header: Stage-Specific Evidence Integration

The system should implement stage-specific integration of structured evidence by comparing variants that calibrate only the central timeline versus both stages, as this analysis reveals where multimodal evidence is most useful. This allows the AI to determine whether structured data is best used to refine the backbone or recalibrate the full trajectory.

  1. Bold Header: Gap Analysis for Missing Clinical Information

The system can perform an auxiliary gap analysis by comparing text-derived timelines against tabular records to identify information that is absent, delayed, or semantically compressed in structured tabular data. This allows the AI to recognize that structured data is not merely a noisier version of the narrative record; it is a different view of the patient trajectory, ensuring that critical details like symptom progression are not overlooked.

  1. Bold Header: Optimized LLM Reasoning for Temporal Tasks

The underlying large language model should be used as an instruction-tuned foundation model within a structured retrieval-augmented pipeline, combining its semantic interpretation of narrative text and flexible temporal reasoning over heterogeneous evidence sources. This leverages the LLM's capabilities while constraining it with structured prompts to ensure accuracy in complex temporal tasks.

Abstract

Clinical language models increasingly operate over electronic health records (EHRs), yet patient records are not stored as temporally grounded trajectories. Clinical notes describe symptoms, assessments, and disease progression, but often compress or narratively reorder events. Structured EHR rows provide timestamps for labs, medications, vitals, and procedures, but capture only part of the clinical story. We formulate clinical timeline reconstruction as retrieval-augmented temporal grounding: constructing a patient trajectory by using narrative text for event semantics and structured rows as partial temporal evidence. We introduce a scaffolded workflow that extracts central narrative events, builds an initial temporal scaffold, attaches non-central events, and calibrates timestamps using retrieved structured EHR rows. We evaluate on 40 discharge summaries, including 15 i2b2-derived and 25 MIMIC-IV summaries, each with manual gold-standard timelines and aligned structured EHR data. Across models, multimodal calibration left event match rates largely unchanged and generally improved temporal performance: mean paired case-level multimodal-unimodal differences were positive in 7 of 12 model-metric comparisons across concordance and AULTC, with none negative. However, uncertainty was substantial given the 40-case sample; paired case-level bootstrap intervals excluded zero only for the DeepSeek V3.2 AULTC improvement. A gap analysis shows that 35.1% of text-derived events have no structured counterpart. These findings support treating structured EHR data as partial temporal evidence for narrative-derived patient trajectories.

Sources

Related papers