MedHorizon: Towards Long-context Medical Video Understanding in the Wild

Long medical video understanding requires models to navigate highly redundant, sparse, and context-dependent evidence distributed across full clinical procedures. MedHorizon introduces an in-the-wild benchmark designed to rigorously test whether multimodal large language models (MLLMs) can perform full-procedure retrieval and multi-hop clinical reasoning using extremely sparse evidence. The best model evaluated reached only 41.1% accuracy, indicating that current systems remain far from robust full-procedure understanding.

How it works

MedHorizon is an in-the-wild benchmark comprising 340 public full-length videos spanning 7 organs and 2 clinical scenarios: diagnostic examination and surgery, totaling 759 hours of clinical video. It provides 1,253 evidence-grounded multiple-choice questions that jointly evaluate sparse evidence understanding and multi-hop clinical reasoning. The benchmark preserves the full procedural stream rather than pre-cut clips, requiring models to search noisy procedural streams before interpreting and aggregating findings.

Benchmark Construction

The benchmark construction follows an evidence grounded pipeline (Fig. 2), which involves several rigorous steps:

1. Source Annotations + Task Templates are collected covering phases, anatomical regions, examination targets, lesion properties, instruments, and procedure level events.

2. Each QA item is generated from explicit video evidence in expert annotations by retrieving evidence from the full video (including sparse key frames and temporally relevant segments), building candidate questions around the annotated event or finding.

3. Candidates are filtered through Automatic filtering to remove temporally invalid, ambiguous, duplicate, or malformed items (removing 81 invalid candidates).

4. Distractors are strengthened using Language rewriting with GPT-5.4 and by drawing distractors from clinically plausible alternatives.

5. The final stage is a Human verification protocol where three non-physician data reviewers audit every retained item against the source annotations to ensure evidence support and label consistency, resulting in 1,253 final QA items.

Diagnostic Findings

The evaluation reveals four key findings regarding model performance:

1. Performance does not scale reliably with more frames, showing non-monotonic frame scaling, where adding frames can introduce near-duplicate anatomy and degrade accuracy once redundant context dominates the useful signal.

2. Evidence retrieval and clinical interpretation remain primary bottlenecks, exposing a dual bottleneck requiring both evidence-aware temporal retrieval and clinically grounded visual verification.

3. There is a weakness in weak procedural reasoning and attention drift under redundancy, where model attention drifts toward visually repetitive but clinically non-decisive frames and patches rather than sparse answer-relevant evidence.

4. Generic sampling methods show limited ability, as they often fail to address the challenge of finding subtle decisive frames and maintaining global procedural continuity or multi-step aggregation.

Task Hierarchy and Bottlenecks

MedHorizon categorizes tasks into fine-grained understanding (PR, SD, RL, BPQ, LLS, Site, Hist, Size) and semantic reasoning (IR, WR, RSR, CPR). The hierarchy requires models to move from one-hop visual-semantic tasks like Instrument Recognition (IR) to three-hop aggregative tasks, such as Count-Proportion Reasoning (CPR) and Lesion Analysis (LA), which demand locating repeated events or lesion-relevant moments and aggregating them into a unified, procedure-level clinical judgment. The results show that semantic reasoning tasks, particularly CPR and LA, remain difficult because they require counting, comparing, and aggregating distributed observations across the same procedure.

Sampling Strategies

The paper analyzes the impact of different sampling strategies on performance. Uniform sampling remains stable by preserving procedural coverage but is evidence-agnostic. Conversely, change-focused samplers improve short-window tasks like RL and Size but hinder workflow-sensitive tasks (e.g., PR). Specialized methods, such as AKS, ViLAMP, and WFS-SB (using 128 frames), show modest gains; for instance, WFS-SB reaches 32.8% on Site, reflecting its design to employ semantic-boundary selection to retain structured temporal context. However, these improvements remain task-dependent and still underperform on tasks requiring global procedural continuity or multi-step aggregation like WR and RSR.

Conclusion

MedHorizon provides a rigorous testbed for MLLMs by explicitly evaluating the full-procedure setting, demonstrating that the primary obstacle is the coupling between sparse evidence localization and cross-temporal reasoning. Future medical MLLMs need stronger mechanisms for sparse evidence perception and contextual reasoning rather than simply scaling context length. The benchmark remains limited by public-video diversity and annotation quality but offers a clear path for research evaluation in this challenging domain.

As a fastidious research assistant, I have analyzed the MedHorizon paper and identified critical weaknesses in current Medical Multimodal Large Language Models (MLLMs) when applied to long-context clinical video understanding.

The core limitation is that current models fail because they treat long medical procedures like general long videos—assuming salient evidence is dense and easy to retrieve—when, in reality, the decisive evidence is extremely sparse, spatially subtle, and temporally distributed across redundant frames.

Here are the specific improvements I propose for AI systems based on the MedHorizon findings:

) 1. Implement Evidence-Aware Temporal Retrieval Mechanisms (Addressing RQ2).

The system must move beyond simple frame sampling or uniform coverage.

• Specific Improvement: Develop a retrieval module that uses clinical context (e.g., during resection phase, post-withdrawal) to guide sparse evidence search, rather than relying on visual saliency alone. This requires integrating procedural knowledge graphs or temporal segmentation models with the vision encoder.

• Improved AI Capability: The system will be able to reliably locate rare, subtle clinical events (e.g., a specific lesion appearance at a precise 15-minute mark) that are visually indistinguishable from redundant frames, significantly improving accuracy in tasks like Lesion Analysis (LA) and Step Discrimination (SD).

) 2. Develop Sparse Evidence Perception and Compression Modules (Addressing RQ4).

The model needs to learn to ignore redundancy while preserving clinically relevant information.

• Specific Improvement: Integrate a mechanism for evidence compression or contextual denoising during the attention aggregation phase. This module should be trained to suppress attention on patches exhibiting high spatial redundancy with keyframes, focusing computational resources only on regions where the visual features deviate significantly from the established procedural norm (i.e., identifying outliers).

• Improved AI Capability: The system will reduce attention drift, allowing it to maintain focus on sparse, decisive moments even when surrounded by highly similar anatomical views. This directly addresses the failure of generic sampling methods, leading to more stable performance across different video inputs.

) 3. Establish Procedural Reasoning and Contextual Binding Layers (Addressing RQ3).

The system must learn the why and where behind observations, not just the what.

• Specific Improvement: Introduce a dedicated reasoning layer that explicitly models clinical workflows (e.g., gastric bypass steps, colonoscopy phases) as structured temporal dependencies. This layer should be tasked with binding localized visual cues (e.g., an instrument) to its correct procedural phase or anatomical region before performing cross-temporal comparisons (RSR).

• Improved AI Capability: The system will transition from pattern recognition to true clinical reasoning. It will be able to correctly answer multi-hop questions that require linking a finding in the initial phase to a subsequent event in a later phase, mimicking the holistic judgment of an experienced clinician.

) 4. Implement Non-Monotonic Frame Scaling Strategies (Addressing RQ1).

The system must stop assuming that more input always equals better performance.

• Specific Improvement: Design an adaptive sampling strategy that dynamically adjusts frame budget based on the identified evidence density and the task complexity, rather than using a fixed cap or uniform sampling. The system should prioritize retaining frames with high information gain for specific reasoning tasks (e.g., prioritizing frames around known procedural transitions).

• Improved AI Capability: The system will achieve robust performance across a wide range of video lengths without suffering catastrophic degradation in accuracy when context length increases, overcoming the context-dilution effect observed in models like MedGRPO.

In summary, the improved AI system will transition from a general video analyzer to a specialized clinical reasoning engine capable of performing surgical or endoscopic review by:

1. Locating needle-in-a-haystack sparse evidence accurately.

2. Filtering out visual redundancy to maintain focus on subtle findings.

3. Applying structured procedural knowledge to connect temporally distant observations into coherent clinical judgments.

Medical multimodal large language models (MLLMs) have advanced image understanding and short-video analysis, but real clinical review often requires full-procedure video understanding. Unlike general long videos, medical procedures contain highly redundant anatomical views, while decisive evidence is temporally sparse, spatially subtle, and context dependent. Existing benchmarks often assume this evidence has already been localized through images, short clips, or pre-segmented videos, leaving the retrieval-before-reasoning problem under-tested. We introduce MedHorizon, an in-the-wild benchmark for long-context medical video understanding. MedHorizon preserves 759 hours of full-length clinical procedures and provides 1,253 evidence-grounded multiple-choice questionsthat jointly evaluate sparse evidence understanding and multi-hop clinical reasoning. Its evidence is extremely sparse, with only 0.166% evidence frames on average, requiring models to search noisy procedural streams before interpreting and aggregating findings. We evaluate representative general-domain, medical-domain, and long-video MLLMs. The best model reaches only 41.1% accuracy, showing that current systems remain far from robust full-procedure understanding. Further analysis yields four key findings: performance does not scale reliably with more frames, evidence retrieval and clinical interpretation remain primary bottlenecks; these bottlenecks are rooted in weak procedural reasoning and attention drift under redundancy, and generic sampling methods only partially balances local detail with global coverage. MedHorizon provides a rigorous testbed for MLLMs that retrieve sparse evidence and reason over complete clinical workflows.