Representation Transfer of Foundation Models for Ultra-Widefield Retinal Imaging

summary

The gist

Despite the widespread adoption of foundation models as feature extractors for medical imaging, relatively little is understood about how different pretraining strategies influence the

In short

The episode discusses a paper on transferring knowledge from general foundation models to ultra-widefield retinal imaging tasks. Hosts analyze how different pretraining strategies, such as supervised learning and self-distillation versus Masked Autoencoder, affect the quality of learned features for medical tasks like diabetic retinopathy grading. The conclusion is that the optimal pretraining method depends on the specific medical task.

Key concepts

Foundation Models: Large AI models used as feature extractors that learn general knowledge from vast amounts of data. The discussion focuses on how this general knowledge transfers to specialized medical imaging tasks like retinal scans.
Representation Transfer: The process of successfully moving or applying the knowledge learned by a foundation model trained on general images to a specific, specialized domain, such as ultra-widefield retinal imaging for disease classification.
Pretraining Strategies: Different methods used to teach the foundation model before it is fine-tuned. The paper compared supervised learning, Masked Autoencoder (MAE), and self-distillation objectives to see which method best creates useful features for medical tasks.
Diabetic Retinopathy Grading: A specific downstream task mentioned in the research where the transferred features were tested. This involves classifying or grading the severity of diabetic retinopathy based on retinal images.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Retrieval-Augmented Generation (RAG) — explained in 16 episodes
Federated Learning — explained in 15 episodes
Interpretability — explained in 15 episodes
Sample Complexity — explained in 15 episodes
Generalization — explained in 13 episodes

This episode discusses

The paper

Representation Transfer of Foundation Models for Ultra-Widefield Retinal Imaging · Read on arXiv

MINGYA ALEXA GONG, DA MA, LOVRE ANTONIO BUDIMIR, IVANA MATOVINOVIC, SVEN LONCARIC, MYEONG JIN JU, YUKUN ZHOU, SIEGFRIED K. WAGNER, PEARSE A. KEANE, AND MARINKO V. SARUNIC

Institute of Ophthalmology, University College London · Wake Forest University School of Medicine, Winston-Salem, North Carolina, USA · Virginia Tech-Wake Forest University School of Biomedical Engineering and Sciences, Blacksburg, Virginia, USA · University of Zagreb Faculty of Electrical Engineering and Computing, Zagreb, Croatia · Department of Ophthalmology and Visual Sciences, University of British Columbia

Despite the widespread adoption of foundation models as feature extractors for medical imaging, relatively little is understood about how different pretraining strategies influence the transferability of learned representations to weakly supervised ophthalmic imaging tasks. We investigate this question in ultra-widefield (UWF) retinal imaging by evaluating foundation model representations within a patch-based multiple instance learning (MIL) framework for disease classification on UWF images. We compare Vision Transformer encoders pretrained with supervised, Masked Autoencoder (MAE), and self-distillation objectives, while keeping the downstream aggregation architecture unchanged. Within a controlled comparison of ViT-B encoders pretrained on ImageNet-1k, the choice of pretraining objective substantially influenced frozen representation transfer, with supervised and self-distillation-based models outperforming MAE. A contemporary DINOv3 model pretrained at a larger scale achieved the strongest overall performance, with a quadratic weighted kappa of 0.863 for five-class diabetic retinopathy grading, comparable with DINOv1. Attention analysis further revealed distinct patch-aggregation behaviours associated with the different pretrained representations, while partial fine-tuning substantially reduced the performance gap for MAE. These findings suggest that pretraining strategy influences both representation transferability and the subsequent aggregation of patch-level evidence within MIL, resulting in differences in downstream classification performance.

DOI: 10.1109/ACCESS.2026.3737380

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Representation Transfer of Foundation Models for Ultra-Widefield Retinal Imaging".

Jane: Despite the widespread adoption of foundation models as feature extractors for medical imaging,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So, we’re diving into this paper today, "Representation Transfer of Foundation Models for Ultra-Widefield Retinal Imaging." It sounds super technical, but the big idea is really about seeing how well these massive AI models can learn from general images and then apply that knowledge to very specific medical pictures like those from an ultra-widefield setup.

Jane: That’s a great starting point, Tom; basically, they are looking at whether the 'brain' of a foundation model can successfully transfer its learning across different visual domains. It’s about how much general knowledge translates into specialized medical insights.

Lu: The authors themselves are from some really strong institutions, including UCL and Wake Forest University School of Medicine, which tells you this research is coming from a place with serious expertise in both deep learning and clinical medicine.

Meng: From my side, I’m just curious how they frame the problem; it sounds like they're tackling a huge challenge because ophthalmic imaging is so diverse compared to standard image analysis.

Lalam: From my perspective as the AI, this paper addresses a critical gap in medical AI where we often rely on transferring representations from natural images, but ophthalmic data is unique and hard to model.

The paper's summary: Tom: Okay, so what’s the actual substance of this "Representation Transfer of Foundation Models for Ultra-Widefield Retinal Imaging" paper? It seems they are really focusing on how different ways of pretraining these foundation models—like using supervised learning versus self-distillation—actually affect the quality of those learned representations when we try to use them for things like disease classification.

Jane: Exactly, Tom; the core summary is that they looked at various pretraining strategies, specifically comparing supervised learning with Masked Autoencoder and self-distillation objectives, and they found a clear winner depending on the task.

Lu: They set up this controlled comparison by using Vision Transformer encoders pretrained on ImageNet-1k to see how those different objectives influence the transferability of the features to tasks like diabetic retinopathy grading.

Meng: I'm interested in what they did with the downstream part; they kept the main aggregation architecture exactly the same and just tested if changing how we started learning didn't break things down later.

Lalam: It’s fascinating because it shows that simply making a model bigger isn't enough; the method of teaching it is actually crucial for getting those high-quality, useful features for medicine.

The paper's improvements: Tom: Now we get into the actual findings, and what they found is pretty exciting because they showed that supervised and self-distillation-based models generally outperformed the Masked Autoencoder approach when transferring their representations to ophthalmic imaging tasks.

Jane: That’s a big deal; it suggests that for structured tasks like five-class diabetic retinopathy grading, those methods provide a more robust foundation than reconstruction methods alone.

Lu: They also pointed out that using larger models pretrained at even greater scales, like DINOv3, yielded the strongest overall performance, which was comparable to earlier models on specific grading tasks.

Meng: But they also showed that while DINOv3 was great overall, its performance varied depending on the specific task; for example, it saw a shift in ranking when testing on a smaller binary DeepDRiD task.

Lalam: That variability tells us that there isn't one universal best approach; the optimal pretraining strategy depends entirely on what you need to classify or detect in the medical image.

Conclusion: Tom: So, to wrap up this discussion on "Representation Transfer of Foundation Models for Ultra-Widefield Retinal Imaging," we’ve seen that the choice of how a foundation model is trained makes a real difference in how well its learned features translate to medical tasks.

Jane: We’ve established that supervised and self-distillation techniques offer stronger performance for classification, while DINOv3 shows incredible potential when scaled up, even if the best strategy shifts based on the exact disease you're looking at.

Lu: The implication here is that we need to stop treating foundation models as one-size-fits-all tools and start tailoring our pretraining objectives to the specific medical data domain we are working in.

Meng: For practical applications, this means if you’re building a system for DR grading, you should probably lean into supervised or self-distillation methods first rather than relying solely on standard MAE pretraining.

Lalam: And for me, the most impactful advancement is recognizing that representation transfer isn't just about bigger models; it’s about intelligently selecting the right training recipe to unlock specialized medical knowledge from general vision models.

More episodes

← Home