From Compound Figures to Medical Multi-image Reasoning: Scaling Multimodal Large Language Models with Biomedical Literature

arXiv:2511.22232 · cs.CV, cs.AI, cs.CL · Submitted 2025-11-27 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "From Compound Figures to Medical Multi-image Reasoning".

Jane: As a meticulous researcher, I have thoroughly analyzed both provided texts from the arXiv paper, "From Compound Figures to Medical Multi-image Reasoning:

Tom: First, who's behind it and why it matters.

Title and authors: Tom: We’re looking at the paper's title again, "From Compound Figures to Medical Multi-image Reasoning: Scaling Multimodal Large Language Models with Biomedical Literature." It really highlights the transition from simple image analysis to a much more complex form of reasoning.

Jane: Exactly, Tom; they’re taking those compound figures—where you see a CT scan next to a pathology report, for example—and teaching the AI how to make sense of that whole scene at once.

Lu: The authors are clearly focused on bridging the gap between general MLLM capabilities and the specific needs of medical multi-image data, which is where the real novelty lies in their approach.

Meng: They're using biomedical literature as a massive source of context, which is smart because it grounds the visual understanding in actual medical knowledge instead of just learning patterns from raw images.

Lalam: And that grounding is what makes it so potent; when you combine visual data with deep medical text context, you get a much more reliable form of understanding that feels genuinely useful in a clinical setting.

The paper's summary: Tom: So, the paper summarizes their solution as developing M3LLM, which is trained by generating high-quality training instructions from over two hundred thirty-seven thousand compound figures along with expert text. That’s a lot of data being used to create these specific learning prompts.

Jane: It’s a sophisticated way to teach the AI; instead of just showing it an image and asking a question, they are essentially giving it step-by-step instructions on how to break down the complex visual information into smaller, manageable pieces.

Lu: The summary emphasizes this five-stage, context-aware instruction generation paradigm, which systematically decomposes the multi-image task using a divide-and-conquer strategy to build composite understanding.

Meng: I’m thinking about that decomposition—they're creating instructions for single sub-image VQA, spatial relationship VQA, text-only QA, and multiple-choice VQA to cover all bases. That level of structured prompting is really what makes the training efficient.

Lalam: It’s this systematic approach to instruction generation that really speaks to me; it shows they aren't just throwing a model at the data but are carefully engineering the learning experience for multi-image comprehension.

The paper's improvements: Tom: The suggested improvements center around this entire instruction generation framework, specifically how they structure the training stages—from initial decomposition to final context refinement.

Jane: They propose a crucial "Leakage-Prevented Context Refinement" stage, which is interesting because it’s designed to make sure the final context provided to the model is holistic rather than just giving away an answer from one specific part of the figure.

Lu: That refinement step sounds like a clever way to ensure that when M3LLM generates instructions, those instructions are truly challenging for the model, focusing on medical background and overall insights instead of answer cues.

Meng: From an engineering view, that stage addresses a real weakness in training: ensuring the data remains informative and not just rote memorization of specific visual features. It makes the learning process more robust for general clinical tasks.

Lalam: I think this focus on holistic insight over specific details is what will translate into better real-world performance, because a doctor doesn't look at one image in isolation when making a diagnosis.

Conclusion: Tom: So, to wrap up, the paper introduces M3LLM as a framework that uses structured instruction generation to scale MLLMs for multi-image reasoning using biomedical literature. It’s about systematically turning complex visual scenes into teachable components.

Jane: Essentially, they show that by decomposing the problem and carefully refining the context, we can train models capable of synthesizing information across multiple images in ways that current single-image models simply cannot achieve.

Lu: This work really sets a high bar for how we can utilize existing biomedical data to create more sophisticated AI systems for complex medical tasks.

Meng: I see the practical implication as a more reliable diagnostic assistant because it handles those multi-modal inputs—like combining an MRI with pathology slides—in a coherent way.

Lalam: For me, the real impact is how this advances our culture in healthcare by providing tools that support clinicians in synthesizing complex patient cases faster and more accurately than ever before.

Zhen Chen, Yihang Fu, Gabriel Madera, Mauro Giuffre, Serina Applebaum, Hyunjae Kim, Hua Xu

Department of Biomedical Informatics and Data Science, Yale School of Medicine, Yale University

cs.CV, cs.AI, cs.CL

Submitted: 2025-11-27

Updated: 2026-09-29

Code: https://github.com/franciszchen/M3LLM

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: As a meticulous researcher, I have thoroughly analyzed both provided texts from the arXiv paper, "From Compound Figures to Medical Multi-image Reasoning: Scaling Multimodal Large Language Models with

Key concepts

Compound Figures
These are single images from medical literature that contain multiple related panels or views. The paper uses these as rich training material because they naturally represent complex, multi-image scenarios found in real clinical settings, helping the model learn how to reason across different visual contexts simultaneously.
M3LLM
Medical Multi-image Multi-modal Large Language Model is the proposed model. It is trained by parsing compound figures and expert text to generate high-quality instructions. This process teaches the model not just to look at one image, but to understand spatial relationships, temporal changes, and cross-modal information within a complex set of images.
Instruction Generation Paradigm
This is the core method used to train M3LLM. It involves a five-stage strategy that systematically decomposes compound figures into smaller reasoning tasks (like spatial or text QA). This structured approach ensures the model builds its understanding step-by-step, moving from simple image analysis to sophisticated composite reasoning.
Leakage-Prevented Context Refinement
This is the final quality control stage where information is consolidated. The key innovation here is actively removing details that would directly answer a sub-image's question. Instead, it focuses on providing holistic medical background, forcing the model to reason based on the entire compound figure rather than relying on specific local answers.

Terminology

Summary

As a meticulous researcher, I have thoroughly analyzed both provided texts from the arXiv paper, From Compound Figures to Medical Multi-image Reasoning: Scaling Multimodal Large Language Models with Biomedical Literature. The synthesis below is designed to be comprehensive, detailed, and accurate, capturing the core problem addressed, the proposed solution (M3LLM), and its key methodological innovations.

This research addresses a critical bottleneck in applying Multi-modal Large Language Models (MLLMs) to real-world clinical workflows. While MLLMs show immense potential in healthcare, existing models are predominantly limited to single-image understanding, severely restricting their utility when dealing with complex, multi-image data inherent in clinical scenarios. The fundamental challenge lies in the scarcity of large-scale, high-quality annotated training data necessary to teach models robust multi-image analysis.

The Proposed Solution: M3LLM and a Novel Instruction Generation Paradigm

To overcome this data limitation, the authors propose a novel framework centered around leveraging license-permissive compound images sourced from biomedical literature as a rich, underutilized training resource. The core innovation is a five-stage, context-aware instruction generation paradigm underpinned by a divide-and-conquer strategy. This strategy systematically decomposes the complex task of multi-image analysis into manageable sub-tasks. By doing so, the model is empowered to move beyond simple single-panel analysis and develop a sophisticated composite understanding, learning the intricate spatial, temporal, and cross-modal relationships that are intrinsically present within these compound figures.

The culmination of this instruction generation process is the development of M3LLM (Medical Multi-image Multi-modal Large Language Model). M3LLM is trained by parsing over 237,000 compound figures alongside their contextual expert text to generate high-quality training instructions.

Detailed Breakdown of the Instruction Generation Paradigm:

The paradigm is structured in distinct stages, each designed to build complexity and refine the model's reasoning capabilities:

  1. Decomposition and Sub-task Generation (Stages 1-3): The initial stages focus on systematically breaking down the compound figure into its constituent parts. This involves generating various types of training samples tailored to specific reasoning needs:

  • Generating instructions for Single Sub-image VQA, ensuring the model retains foundational image analysis skills.

  • Generating instructions for Spatial Relationship VQA, converting raw positional data (e.g., from bounding boxes) into structured Q&A pairs to explicitly train spatial reasoning.

  • Generating instructions for Text-only QA, focusing on linguistic and knowledge-based reasoning derived from captions and medical literature summaries.

  • Generating instructions for Multiple-choice VQA, which demands the creation of three plausible distractors to ensure high question quality.

  1. Context Refinement (Stage 5): The final stage, Leakage-Prevented Context Refinement, is a crucial quality control step. It is designed to revise and consolidate information into a single, improved context for the compound image. Critically, this refinement process actively excludes any specific details that directly appear in the answer of a particular sub-image. Instead, it focuses on providing relevant medical background and holistic insights about the entire compound figure, ensuring the resulting context is neutral yet challenging for the model to solve.

Experimental Validation and Superior Performance:

The efficacy of M3LLM is rigorously validated through comprehensive benchmarking:

  • Benchmarking: The authors constructed PMC-MIBench, a benchmark specifically designed for composite understanding, which has been manually validated by medical experts.

  • Comparative Results: Experiments demonstrate that M3LLM significantly surpasses both general-purpose and specialized medical MLLMs across a wide spectrum of scenarios, including multi-image analysis, single-image tasks, text-only reasoning, and multiple-choice questions.

  • Clinical Generalization: Most notably, M3LLM exhibits exceptional generalization to real-world clinical settings. It achieved superior performance on longitudinal chest X-ray analysis using the MIMIC dataset, a key indicator of its practical applicability in clinical diagnostics.

Conclusion and Significance:

This work establishes a scalable and highly efficient paradigm for developing next-generation medical MLLMs capable of composite reasoning across complex multi-image scenarios. By successfully bridging the gap between vast biomedical literature (as data) and real-world clinical applications (as performance metrics), this research offers a robust solution for understanding and reasoning over medical multi-images. M3LLM represents a significant step forward in medical AI, setting a new benchmark for multimodal systems capable of integrating spatial, temporal, and cross-modal information to transform diagnostic workflows and ultimately improve patient care. Its scalable, cost-effective, and clinically relevant framework holds the potential to revolutionize how complex medical data is interpreted.

Improvements for AI systems

Here are specific, actionable improvements for AI systems based on the M3LLM framework described in this paper:

) 1. Transition from Single-Image Focus to Composite Clinical Reasoning:

The core improvement is moving medical LLMs beyond single-image analysis (a current limitation) to genuine multi-image synthesis.

  1. System Capabilities of the Improved AI System (M3LLM):

The improved system, M3LLM, will be able to perform complex diagnostic and prognostic reasoning by:

  • Integrating information across multiple modalities (e.g., MRI + PET + Histopathology) simultaneously to form a unified diagnosis.

  • Analyzing longitudinal data by comparing sequential scans (e.g., chest X-rays) to track disease progression (improvement, deterioration, or stability).

  • Synthesizing spatial relationships between sub-images (e.g., determining the relative position of an MRI slice versus a PET scan) to understand complex anatomical contexts.

  1. Key Technical Improvements Driving System Capability:

The system's enhanced capability is driven by the novel five-stage, context-aware instruction generation paradigm:

  • Instead of relying on simple image-text pairing, the system is trained on structured instructions derived from compound figures that explicitly teach it to perform divide-and-conquer reasoning.

  • The stages ensure comprehensive medical grounding: Stage 1 (Summarization) provides clinical context; Stage 2 (Knowledge Complementation) enriches the instruction with domain-specific pathophysiology; and Stage 3 (Visual Perception Enhancement) ensures precise, sub-image level visual feature extraction.

  • Stage 4 generates diverse reasoning tasks—including multi-image VQA, single-image VQA, text-only QA, and structured multi-choice VQA—ensuring the model is robust across all necessary clinical interaction formats.

  • Stage 5 (Leakage Prevention) ensures the training data remains challenging by removing answer leakage cues, leading to more reliable and clinically relevant reasoning.

  1. Specific Performance Gains:

The improved M3LLM is expected to show:

  • Significant superiority in multi-image VQA tasks on benchmarks like PMC-MI-Bench (achieving 15.0 BLEU@4 vs. state-of-the-art).

  • Enhanced accuracy in longitudinal clinical validation tasks (e.g., MIMIC X-ray analysis), demonstrating superior ability to track disease progression compared to baselines by detecting subtle spatiotemporal relationships.

  • Robust performance across diverse medical specialties (BMS, CM, DLM, etc.) and imaging modalities (CT, MRI) on public benchmarks like OmniMedVQA and MMMU-Med.

  1. Clinical Application Impact:

The resulting AI system can be deployed as a clinical decision support tool to:

  • Aid radiologists in synthesizing complex multi-panel studies (e.g., integrating CT, MRI, and pathology slides) for faster, more accurate diagnosis of subtle pathologies like liver masses or lung nodules.

  • Support longitudinal patient monitoring by automatically assessing disease progression from serial imaging studies.

Sources

Related papers