Mr3D-VL: A generalist vision language foundation model for Multiparametric 3D Magnetic Resonance Imaging

Mr3D-VL is a dedicated visual-language foundation model for multi-parametric 3D magnetic resonance imaging (mpMRI), designed to address critical limitations in current AI models for brain tumor diagnosis and treatment. The paper states: "Multi-parametric magnetic resonance imaging (mpMRI) is a cornerstone for brain tumor diagnosis and treatment, yet current AI models face critical limitations: their lack of natural language interaction and interpretability impedes spatial information integration and cross-modal reasoning required clinically."

The model has 4 billion parameters and employs an unsupervised pre-trained shared 3D encoder and 4D rotational positional embedding for dual modality-spatial integration. Its cross-modal projection layer uses a multi-resolution feature implantation strategy to enhance feature perception across resolutions. The paper reports: "Experimental results show significant improvements over existing 4B/7B/30B domain-specific and general-purpose models in text generation tasks, achieving a BERTScore of 0.856 for report generation, with question-answering accuracy at 0.713 and multiple-choice accuracy at 0.912."

The key challenges addressed include: "(1) substantial differences in tissue properties reflected by different modalities (e.g., T1, T2, DWI), necessitating models to understand the physical meanings and clinical values of each modality; (2) spatial misalignment caused by tissue displacement due to voluntary/involuntary patient motion across modalities with varying scan times; and (3) the need for cross-modal associative reasoning in clinical queries."

The paper's key contributions are:

1. "LLMs-Driven Data Generation: Leveraging large language models (LLMs) to parse clinical reports and integrate inference results from well-established smaller models (e.g., segmentation, detection), this approach generates diverse multimodal image-text datasets for cross-modal learning, addressing challenges of data scarcity and semantic incoherence."

2. Unsupervised Pre-training with Shared Encoder: A modality-agnostic 3D visual encoder is shared across all imaging modalities, with unsupervised pre-training implemented via the Dino-v2 architecture.

3. "4D Rotational Positional Encoding: Incorporating structural characteristics of 3D volumetric data, a 'depth' dimension is introduced to traditional 3D positional encoding (time, width, height), constructing a 4D coordinate system (time, depth, width, height)."

4. Multi-Resolution Image Feature Implantation Strategy: Low-, medium-, and high-resolution feature maps are synchronously extracted from the pyramid-style feature extraction architecture of the visual encoder.

The model architecture follows a three-module design: a vision encoder, an MLP-based vision language projection, and a large language model (LLM). The LLM backbone is Qwen3-4B-Instruct, and the visual encoder uses MedNext with DINO-v2 unsupervised pre-training. The paper notes: We select MedNext as the backbone network for the visual encoder, leveraging its innovative designs such as the ConvNeXt3D architecture, UpKern technology, and composite scaling strategies.

The training pipeline consists of pre-training and post-training stages. Pre-training has three phases: Stage S0: Visual-Language Alignment (training only the projection module), Stage S1: Unimodal Pre-training (unfreezing all components), and Stage S2: Multimodal Pre-training (mixed-modal inputs). The dataset comprises 103,605 cases of brain multiparametric magnetic resonance imaging (mpMRI), totaling 460,541 images, with an average of 4.4 modalities per case.

Post-training includes supervised fine-tuning with nearly 300,000 samples consisting of "3D MR imaging report generation data and Question-and-Answer (Q&A) data," with Q&A further divided into open-ended and multiple-choice questions.

Experimental results show Mr3D-VL's superiority across all tasks. For report generation, it achieved BLEU-4 (0.169) and ROUGE-L (0.353) with BERTScore (0.856), METEOR (0.496), and CIDEr (0.655). For open-ended question answering, it achieved BLEU-4 (0.525), ROUGE-L (0.738), BERTScore (0.794), METEOR (0.747), and CIDEr (0.837). Accuracy results show Open-ended question accuracy: 0.713 (+20% improvement over existing models) and Multiple-choice question accuracy: 0.912 (+13% improvement over existing models).

The paper also demonstrates efficiency advantages: For 5-modal input, MR3D achieves 96% reduction in FLOPs (2,064.64 G vs. 57,186.88 G for Qwen3.5-4B) and peak usage of 9,215.77 MB under 5-modal input- 93% lower than Lingshu-7B (115,741.43 MB) and 86% lower than Hulu-med-7B (66,181.22 MB).

The paper concludes: "Mr3D-VL achieves breakthroughs through four key innovations: (1) High-quality multimodal data synthesis; (2) Modality-adaptive feature extraction; (3) 4D rotary positional encoding (4D-RoPE); (4) Cross-modal alignment optimization. Experimental evaluations demonstrate that Mr3D-VL outperforms general-purpose LLMs and 2D vision-language models (VLMs) in multimodal diagnostic accuracy, with stable performance across varying input modalities."

Improvements to AI Systems:

1. Unsupervised Modality-Agnostic 3D Encoder with DINO-v2 Pre-training

• Implement a shared 3D visual encoder (e.g., MedNext) pre-trained unsupervised via DINO-v2, eliminating the need for labeled imaging data during initial feature learning.

• This enables the AI to extract robust, transferable volumetric features across T1, T2, DWI, and other modalities without manual annotation, improving generalization to unseen MRI protocols or rare tumor types.

2. 4D Rotational Positional Encoding (4D-RoPE) for Spatial-Temporal Alignment

• Extend standard 3D positional encoding (time, width, height) with a depth dimension, creating a 4D coordinate system (time, depth, width, height).

• This allows the AI to explicitly model tissue displacement and misalignment across modalities caused by patient motion, improving cross-modal spatial reasoning and reducing errors in tumor boundary detection or lesion localization.

3. Multi-Resolution Feature Implantation Strategy

• Synchronously extract low-, medium-, and high-resolution feature maps from a pyramid-style visual encoder and fuse them via a cross-modal projection layer.

• This enhances the AI’s ability to perceive both fine-grained details (e.g., small lesions) and global anatomical context, improving diagnostic accuracy for heterogeneous tumors and reducing false negatives in small or diffuse abnormalities.

4. LLM-Driven Synthetic Data Generation for Multimodal Learning

• Use LLMs to parse clinical reports and integrate outputs from smaller models (segmentation, detection) to generate diverse, semantically coherent image-text pairs.

• This addresses data scarcity and label inconsistency, enabling the AI to train on large-scale, varied multimodal datasets without manual annotation, thereby improving robustness to rare conditions and diverse clinical query phrasings.

5. Three-Stage Pre-training Pipeline (Alignment → Unimodal → Multimodal)

• First, train only the projection module for visual-language alignment; then unfreeze all components for unimodal pre-training; finally, use mixed-modal inputs for multimodal pre-training.

• This staged approach stabilizes learning, prevents catastrophic forgetting, and ensures the AI gradually integrates visual and textual knowledge, leading to better cross-modal reasoning and higher accuracy in open-ended and multiple-choice questions.

6. Cross-Modal Projection with Multi-Resolution Fusion

• Replace simple linear projections with a multi-resolution feature implantation layer that fuses features at different scales before feeding into the LLM.

• This improves the AI’s ability to answer clinical queries requiring both local (e.g., tumor size) and global (e.g., midline shift) information, boosting performance on tasks like report generation and structured Q&A.

7. Efficient 3D Volumetric Processing

• Adopt the model’s architecture (4B parameters, shared encoder, 4D-RoPE) to reduce FLOPs by 96% and memory usage by 86–93% compared to existing 7B/30B models.

• This enables deployment on standard clinical hardware (e.g., single GPU with 9GB memory) for real-time 3D MRI analysis, making advanced diagnostic AI accessible in resource-limited hospitals.

What the Improved AI System Can Do:

• Generate accurate, free-text radiology reports from multi-parametric 3D MRI (T1, T2, DWI, etc.) with BERTScore 0.856, outperforming 7B/30B general-purpose models.

• Answer open-ended clinical questions (e.g., “What is the likely grade of this tumor?”) with 71.3% accuracy, a 20% relative improvement over existing systems.

• Solve multiple-choice diagnostic questions (e.g., “Which sequence best shows peritumoral edema?”) with 91.2% accuracy, a 13% improvement.

• Handle spatially misaligned multimodal inputs (due to patient motion) by explicitly modeling 4D coordinates, ensuring consistent tumor segmentation and feature extraction across scans.

• Learn from unlabeled 3D MRI data via DINO-v2 pre-training, reducing dependence on expert annotations and enabling rapid adaptation to new imaging protocols.

• Process up to 5 modalities simultaneously with minimal computational overhead (2,064 G FLOPs vs. 57,187 G for Qwen3.5-4B), enabling real-time clinical decision support.

• Generate diverse synthetic training data from existing reports and segmentation outputs, allowing continuous model improvement without new manual labeling.

Multi-parametric magnetic resonance imaging (mpMRI) is a cornerstone for brain tumor diagnosis and treatment, yet current AI models face critical limitations: their lack of natural language interaction and interpretability impedes spatial information integration and cross-modal reasoning required clinically. Key challenges arise from significant physical meaning differences across modalities, spatial misalignment due to scan intervals, and the need for complex multi-feature interpretation in tasks like glioma grading. While visual-language models (VLMs) show promise in cross-modal understanding, existing methods focus mainly on 2D image modeling, neglecting direct perception of 3D volumetric space. Although 3D VLMs have been proposed for report generation and feature alignment in 3D CT imaging, mpMRI applications demand collaborative inference across multiple imaging modalities-a requirement unmet by current solutions. To address this, we introduce Mr3D-VL, a dedicated visual-language foundation model for multi-parametric 3D MRI. With 4 billion parameters, it employs an unsupervised pre-trained shared 3D encoder and 4D rotational positional embedding for dual modality-spatial integration. Its cross-modal projection layer uses a multi-resolution feature implantation strategy to enhance feature perception across resolutions. Experimental results show significant improvements over existing 4B/7B/30B domain-specific and general-purpose models in text generation tasks, achieving a BERTScore of 0.856 for report generation, with question-answering accuracy at 0.713 and multiple-choice accuracy at 0.912.