Medical foundation models converge less under label supervision

As a fastidious and diligent AI researcher, I have meticulously analyzed the provided excerpts from arXiv to construct a comprehensive summary of the research findings regarding representational convergence in medical imaging encoders. My analysis focuses on isolating the core arguments, experimental setups, and conclusions presented in both segments to provide a detailed and accurate description of this work.

Here is the detailed synthesis:

This research investigates the mechanisms that drive representational convergence among different image encoders (e.g., Vision Transformers) when trained on medical data, specifically contrasting the influence of different training objectives—namely, self-supervision versus clinical/label supervision. The central thesis is that the training objective, particularly self-supervision, is the primary driver of alignment among encoders, rather than clinical supervision or model scale.

The authors employed a rigorous controlled experimental setup to isolate the effect of the training objective:

1. Controlled Training Matrix: They trained 12 different image encoders (varying only in objective, backbone capacity—ViT-S or ViT-B—and modality—chest radiography or histopathology) while holding the training budget fixed and using a single seed per cell. The objectives varied across three main types:

• Masked Image Self-Supervision (the self-supervised baseline).

• Clinical Label Supervision.

• Image-Text Contrastive Learning.

2. Convergence Measurement: Alignment among these encoders was measured using a pairwise alignment defined by the provided methodology, tested across a large pool of chest radiographs (650,982) and other imaging pools. The results were analyzed via bootstrapping over shared evaluation cases to account for sample variability rather than training seed variability.

3. Orthogonal Metric: The analysis was also performed using the orthogonal Procrustes metric to ensure robustness against specific geometric transformations.

The experiments yielded several critical findings that directly challenge prior assumptions in the field:

• Objective Dictates Alignment: The most significant result is that the training objective, specifically self-supervision, is what aligns these encoders. This finding separates two previously entangled ideas:

• The original hypothesis suggested convergence was driven by scale and downstream performance, treating the objective as largely indifferent.

• This study demonstrates that adding a clinical label or report objective to an otherwise identical backbone moves its representation away from, and not toward, its peers.

• Self-Supervision Superiority: When data and initialization are controlled (shared data and initialization removed), the synthetic model confirms the controlled experiment: the masked-reconstruction objective yields a more convergent global representation than the label-prediction objective, at every level of label informativeness.

• The Supervised-Minus-Self-Supervised Gap: The gap between supervised and self-supervised performance remained consistently negative across all tested levels of label informativeness (p < 0.0001). Crucially, this gap did not close as the labels became more informative (Fig. 4b–d; Supplementary Table 4), even when using the orthogonal Procrustes metric.

The research also systematically tested whether convergence scales with conventional metrics:

• Lack of Scaling: Convergence does not scale with model size, downstream performance, or recency. Relating the residual distance to these factors across 18 encoders (ranging from 7 million to 1.1 billion parameters) showed no significant relationship with model size (Spearman r = 0.302, p = 0.223), linear classifier AUROC (0.131, p = 0.604), or release year (0.206, p = 0.412).

• Weak Shared Geometry: Consequently, the shared geometry between encoders is weaker and more useful than prevailing accounts predict. The claim that image and language encoders meet in a shared structure was not supported by their cross-modal alignment measurements (Fig. 7e), which showed alignment at floor level and falling toward zero as the evaluation pool grew (from 1,000 to 20,000 cases).

Despite the weaker geometric alignment across modalities and encoders, the authors derive a highly consequential practical message:

• Functional Interchangeability is Possible: Weak geometric alignment is still sufficient for functional interchangeability. A diagnostic classifier trained in one encoder's coordinate system can be transferred to other encoders and external sites, provided a common coordinate system (or learned affine map) is established.

Here are the specific improvements to AI systems derived from this scientific paper, categorized by the capability they enable:

) Improvements for Model Training and Objective Design:

1. Improve foundation model alignment by prioritizing a robust, self-supervised pretraining objective over clinical supervision or image-text objectives when aiming for general representational convergence.

2. Design objective-driven training matrices that explicitly vary the training objective (self-supervised, label-supervised, image-text) while holding data, architecture, and scale fixed to isolate the causal driver of alignment.

3. Utilize synthetic generative models with a clinically supervised signal subspace to train encoders under controlled objectives to ensure that convergence reflects the preservation of meaningful clinical structures rather than superficial statistical noise.

) Improvements for Model Deployment and Interoperability:

1. Enable true encoder-agnostic deployment by implementing a shared, anchor-based coordinate system (relative representations) for all medical encoders and their downstream classifiers.

2. Develop linear feature stitching techniques that learn affine maps to transfer knowledge between different frozen encoders, allowing a diagnostic classifier trained in one encoder’s space to be applied effectively across all others without retraining.

3. Create deployable tools that rely on shared latent coordinates rather than requiring perfect representational convergence across all models, enabling functional interchangeability even when geometric alignment is modest (e.g., cross-site transfer).

) Improvements for Clinical Tool Validation and Trust:

1. Implement rigorous drift detectors based on per-case cross-encoder neighbor disagreement as an unsupervised score to monitor distribution shifts in deployment, providing a definitive negative result for the current panel regarding this specific detection method.

2. Ground automated diagnostic readouts against expert reader judgments using a consensus configuration that recovers clinical co-occurrence structure (e.g., inter-finding distances correlated with comorbidity matrices) rather than relying solely on learned classification scores to predict case-level clinical similarity, which proved unreliable for expert perception tasks.

3. Ensure subgroup robustness by applying subgroup-specific checks when relying on consensus summaries, explicitly caution clinicians that these representations are least reliable for under-represented patient populations where data is sparse.

) Improvements for Clinical Application and Performance:

1. Enhance diagnostic performance in clinical settings by leveraging the learned shared geometry to perform cross-encoder transfers across different imaging modalities (e.g., transferring a CXR classifier to a histopathology encoder) while retaining over 85% of within-encoder oracle performance, even with modest alignment.

2. Increase the reliability of diagnostic tools by focusing on per-finding alignment metrics (mKNN) correlated with finding prevalence and learnability, which more accurately reflect how the shared geometry organizes disease in a clinically meaningful way, rather than relying on overall pool-level convergence which is sensitive to pool size.

3. Develop methods for better handling of modality gaps by analyzing cross-modal alignment decay as a function of evaluation pool size, providing clear evidence that image and language encoders do not converge to a shared structure across modalities when tested against realistic clinical data pools.

This improved AI system can:

1. Perform reliable, encoder-agnostic diagnostic classification using any trained medical encoder from the panel, regardless of its specific pretraining objective (SSL, supervised, or image-text).

2. Transfer a diagnostic model trained on one imaging modality (e.g., Chest Radiography) to another modality (e.g., Histopathology) by fitting an affine map and applying the transferred classifier head to the target encoder's features, achieving high performance retention (median 85.3% AUROC).

3. Develop a drift-aware deployment pipeline for clinical tools that can detect distribution shifts based on per-case neighbor disagreement, providing a robust safety mechanism during real-world deployment.

4. Generate consensus geometries that accurately reflect the co-occurrence of medical findings (comorbidity structure) rather than administrative coding taxonomies (ICD-10), making the resulting representations clinically relevant for patient stratification and prognosis.

5. Provide a principled framework for assessing representational convergence, shifting the design paradigm from seeking universal models to optimizing for specific training objectives that yield the most useful, albeit modest, shared geometry.

Diagnostic classifiers and imaging biomarkers are fitted on the embeddings of medical foundation models. These models are replaced as new versions appear. This practice assumes that different models represent images alike. We tested this assumption with more than 750,000 images from 14 datasets in five imaging modalities, 18 public models, and 101 models trained on chest radiographs and histopathology that differ in pretraining objective, label type, size, random seed, initialization, or training patients. Agreement was measured as the mutual k-nearest-neighbor overlap, the share of an image's nearest neighbors common to two models. Two public models shared on average 0.091 of the 10 nearest neighbors of a chest radiograph. A randomly initialized network shared 0.036 with them. In the four other modalities, they shared at most 0.300. Convergence depended on the pretraining objective. Models trained with label supervision converged least in all five controlled settings. On chest radiographs, two label-supervised models that differed only in their random seed shared 0.062 to 0.100 of their nearest neighbors. Two models trained with self-distillation, masked image modeling, or contrastive pretraining shared 0.411 to 0.802. On chest radiographs, agreement depended more on the source dataset and the radiographic view than on the findings. It was not significantly associated with accuracy. A linear mapping between the embeddings of two models, fitted on 4,096 unlabeled radiographs, transferred classifiers for 14 chest findings with 0.987 of their original area under the receiver operating characteristic curve. Convergence can therefore be chosen when a model is trained. A chest radiograph classifier can be transferred to a new model without new labels.