LFM: Leveraging Foundation Models for Source-Free Universal Domain Adaptation

summary

The gist

The gist: A framework leveraging foundation models for source-free universal domain adaptation adapts a pre-trained source model to an unlabeled target domain under both covariate and label shifts

In short

This framework adapts a pre-trained model to an unlabeled target domain without source data by using foundation models (LFM). It uses a vision-language model to generate pseudo-labels and determines label shift types via similarity scores. A consensus strategy refines these labels using the target model initialized with the source knowledge, leading to superior adaptation performance across various label shifts.

Key concepts

Source-free universal domain adaptation (SF-UniDA): This is a method that adapts a model trained on one dataset (source) to work well on a completely different, unlabeled dataset (target). The key challenge is doing this without ever seeing any data from the original source domain, which requires sophisticated techniques to handle shifts in data distribution and class labels.
Foundation Models (FMs): These are large, powerful models like vision-language models that have been trained on vast amounts of data. They are known for their strong general knowledge and ability to perform tasks with very little or no specific training for a new task, making them ideal tools for generating useful predictions in adaptation scenarios.
Pseudo-label refinement (PLR): This module takes the initial predictions (pseudo-labels) generated by the foundation model and improves their quality. It uses the target model, which starts with knowledge from the source domain, to correct or align these initial labels. This consensus process ensures that the final labels are both semantically sound and consistent with what is learned from both domains.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 18 episodes
Federated Learning — explained in 17 episodes
Retrieval-Augmented Generation (RAG) — explained in 16 episodes
Interpretability — explained in 15 episodes
Quantum Fisher Information (QFI) — explained in 15 episodes

This episode discusses

The paper

LFM: Leveraging Foundation Models for Source-Free Universal Domain Adaptation · Read on arXiv

Jing Li, Pan Liu, Meng Zhao, Wanli Xue, Yanhong Yang, Xu Cheng, Fan Shi, Jianhua Zhang, Qinghua Hu Senior Member, IEEE and Shengyong Chen Senior Member, IEEE

Tianjin University of Technology Department of Computer Science and Engineering School of Artificial Intelligence Engineering Research Center of Learning-Based Intelligent System Ministry of Education of the People’s Republic · Engineering Research Center of City Intelligence and Digital Governance Ministry of Education’s Republic

Source-free universal domain adaptation (SF-UniDA) adapts a pre-trained source model to an unlabeled target domain under both covariate and label shifts, without access to source data. However, existing SF-UniDA methods rely on inefficient techniques such as threshold tuning and clustering. Foundation models (FMs), known for their generalization and zero-shot capabilities, remain underexplored in SF-UniDA. In this paper, we propose a framework that leverages foundation models (LFM) for SF-UniDA. We use a vision-language model (VLM) to compute similarities between target samples and text labels, including those for unknown classes generated by prompting a large language model. The label shift type is determined by analyzing the coefficient of variation of a similarity-based sample-level score. Unknown samples are identified using a binary Gaussian mixture model fitted to another similarity-based metric. Under a consensus strategy, the pseudo-labels generated by the VLM are refined by the target model initialized with the pre-trained source model, integrating knowledge from both the source domain and foundation models. Finally, these refined pseudo-labels are used to train the target model. Extensive experiments across all possible label shifts and multiple benchmarks demonstrate the effectiveness and superiority of our proposed LFM framework. Our code is available at https://github.com/iamjingli/LFM.

DOI: 10.1109/TMM.2026.3730952

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "LFM: Leveraging Foundation Models for Source-Free Universal Domain Adaptation".

Jane: The gist:

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Before we get into the details of LFM: Leveraging Foundation Models for Source-Free Universal Domain Adaptation, I want to talk about what this title actually suggests. It’s not just about adapting a model; it’s specifically about doing that without any data from the source domain.

Jane: That's the crucial constraint there. It means the method has to be entirely self-contained and rely on external knowledge, which is where foundation models come in handy, right?

Lu: Right. They are moving away from methods that just use simple threshold tuning or clustering for those unknown cases, which they say are inefficient techniques.

Meng: What I mean is they aren't just throwing a heuristic at it; they're trying to build a systematic way to handle the label shift and the unknown classes simultaneously under that strict source-free rule.

Lalam: The paper argues that foundation models, especially VLMs like CLIP, offer semantic priors that are much better than task-specific classifiers when you have this kind of distribution mismatch.

Tom: So what they propose is a framework where the VLM computes similarities between target samples and text labels, and then they use that similarity score to figure out the type of label shift happening.

Jane: And instead of just guessing how it’s shifted, they analyze the coefficient of variation of those sample-level scores to determine if it's OSDA or OPDA.

Lu: If it is OSDA or OPDA, they use a binary Gaussian mixture model, or GMM, fitted to another similarity-based metric to identify which samples are actually unknown categories.

Meng: That’s an improvement over just setting a fixed threshold; they’re using the data distribution itself to find the unknown ones.

The paper's summary: Tom: So, summarizing what LFM does, it starts with generating text labels for unknown categories using an LLM, then using a VLM for zero-shot inference on both known and those generated labels.

Jane: Then they introduce a sample-level metric score based on the similarity between the image and the text label. That score is what they use to figure out what kind of label shift they are dealing with.

Lu: They analyze the coefficient of variation of these scores across all target samples to determine if it's OSDA or OPDA, which dictates how they proceed next.

Meng: If it’s OSDA or OPDA, they use that binary GMM on another similarity metric to separate the known samples from those that are truly unknown categories in the target domain.

Lalam: If the shift isn't OSDA or OPDA, they just take the pseudo-labels directly from CLIP, which is a simpler path for those scenarios.

Tom: But where it gets interesting is when there's a mismatch, because they design a module where the target model, starting from its source model weights, refines those initial pseudo-labels through a consensus strategy.

Jane: That refinement process integrates knowledge from both the foundation models and that pre-trained source model simultaneously to get better labels.

Lu: They call this pseudo-label refinement PLR, and it’s designed to be an error control mechanism that enforces consistency between model confidence and semantic consensus.

The paper's improvements: Tom: Now let's talk about the specific modules they introduce that make this framework better than previous source-free universal domain adaptation methods. They focus on systematically addressing both label shifts and unknown-class challenges under the source-free constraint, which is a big step.

Jane: The key improvement seems to be how they handle the shift detection and then how they refine those initial pseudo-labels using that consensus strategy.

Lu: They use a sample-wise metric score, but instead of just looking at one number, they look at its coefficient of variation across all target samples to infer the exact type of label shift.

Meng: And when it's OSDA or OPDA, they replace manual thresholding with that binary GMM fitted to another similarity-based metric to identify those unknown samples.

Lalam: And for the refinement part, they specifically design PLR to incorporate knowledge from both the foundation models and the pre-trained source model into the pseudo-labels provided by CLIP.

Tom: The results show that this PLR module makes modest improvements in OSDA and OPDA settings, but it brings significantly larger gains when the label shift is PDA.

Jane: Why is PDA so much better? Because in the PDA scenario, all target categories are already present in the source domain, so the knowledge transferred from the source model to refine those labels is crucial for accuracy.

Conclusion: Tom: So to wrap up on LFM: Leveraging Foundation Models for Source-Free Universal Domain Adaptation, it’s a structured way to handle label shifts and unknown classes without any source data by using foundation models as semantic priors.

Jane: It moves beyond just using FMs for classification and instead uses them as tools to generate and refine pseudo-labels under distribution mismatch, which is a significant step toward real-world adaptation.

Lu: The main implication here is that systematically modeling label shifts and unknown categories through these modules provides a much more robust approach than relying on heuristic thresholding or simple clustering techniques.

Meng: From an engineering standpoint, the refinement module PLR shows that you can get substantial performance gains when you correctly model the specific shift type, like in the PDA case.

Lalam: The paper gives us a concrete way to use foundation models not just as classifiers but as structured priors that help stabilize and improve pseudo-label generation when data is scarce or mismatched.

Tom: So there it is, LFM: Leveraging Foundation Models for Source-Free Universal Domain Adaptation. We've seen how they use VLMs and LLMs together to create a more systematic approach to this difficult adaptation problem.

Jane: It’s a solid piece of work that shows how foundation models can be integrated into these hard domain adaptation tasks in ways we hadn't really explored before.

More episodes

← Home