ProtoDCS: Towards Robust and Efficient Open-Set Test-Time Adaptation for Vision-Language Models

arXiv:2602.23653 · cs.CV, cs.AI · Submitted 2026-02-27 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "ProtoDCS: Towards Robust and Efficient Open-Set Test-Time Adaptation for Vision-Language Models".

Tom: Large-scale Vision-Language Models (VLMs) face significant challenges in real-world deployment due to distribution shifts,

Jane: First, who's behind it and why it matters.

Title and authors: Tom: So, Jane, we're diving into this paper now titled "ProtoDCS: Towards Robust and Efficient Open-Set Test-Time Adaptation for Vision-Language Models." We’ve talked about how existing methods struggle when they see both known shifted data and completely new data in the test stream.

Jane: Right, Tom, and the core idea is that current approaches often use hard thresholds or entropy minimization, which we know can be brittle when dealing with ambiguous samples. This paper proposes something different to handle those tricky open-set scenarios.

Lu: What I find really compelling about this work is their systematic approach to separating the csID and csOOD samples using a probabilistic mechanism rather than just a single hard cut-off point. It sounds like they're tackling the fundamental problem of sample discrimination in real-world deployment.

Meng: From an engineering standpoint, my main concern with these systems is always computational cost, especially when dealing with large vision-language models. I wonder how this prototype-level update strategy actually manages to be efficient without slowing down inference too much.

Lalam: I think the most impactful aspect for our culture and future deployment is how this framework ensures we don't just adapt to the obvious data but keep the model calibrated and safe when facing uncertainty. It moves us toward more reliable AI in complex environments.

Tom: Exactly, Lu, that separation mechanism sounds smart because it uses a dual-check process to filter samples before any adaptation happens, which seems much safer than one single check. Jane, can you explain what that dual-check looks like in practice?

Jane: Certainly, Tom. The paper describes a four-stage process for an incoming test sample. First up is the First-Check stage where they compute an initial openness score to segregate samples using two different thresholds to build a diversity-aware visual cache and identify trustworthy samples, which is quite systematic.

Lu: And then after that filtering, they move into the Evidence-driven Temporary Prototype Generation stage where the adaptation happens, but only on those trusted samples. That staged approach seems designed specifically to minimize the risk of introducing bad gradients from unknown data into our model's core understanding.

Meng: I need more detail on how they handle the uncertainty during that adaptation phase. If we're updating prototypes, we need a way to know if those updates are actually beneficial or just noise, and I’m looking for something more robust than simple entropy minimization.

Title and authors: Lalam: That's where the uncertainty-aware loss comes in; it explicitly models both aleatoric and epistemic uncertainty, which means the system knows whether its prediction is uncertain because the data itself is noisy or because the model simply doesn't know enough about that specific input.

Tom: That’s a big step away from just minimizing entropy, Jane. It suggests we can actually calibrate our adaptation based on how sure we are about what we're seeing, and I think that directly addresses the overconfidence issue they pointed out in previous work.

Jane: Precisely, Tom. The Final-Verification stage then uses a Gaussian Mixture Model to give us a probabilistic separation between the csID and csOOD samples based on their openness scores, which is much more sophisticated than just using fixed similarity metrics or simple confidence differences.

Lu: That GMM verification is what I find most interesting because it models the distribution of openness scores as coming from two underlying distributions, allowing for a probabilistic decision rather than a binary yes or no answer, which makes the separation much more nuanced.

Meng: But what about the computational side again? The paper states they keep the VLM backbone frozen and only update learnable residual matrices, which seems like a clever way to bypass the need for expensive gradient backpropagation through the whole network during adaptation.

Lalam: That lightweight approach is crucial for practical application; if we can do adaptation without touching all billions of parameters, it makes real-time deployment on edge devices much more feasible and less computationally demanding.

Tom: So, to summarize the core improvements they propose: they replace brittle hard thresholding with a probabilistic GMM verification, use an uncertainty-aware loss instead of just entropy minimization for safety, and keep the backbone frozen for efficiency. Jane, do you have any thoughts on how these combined mechanisms work together?

Jane: They work in sequence to create a cohesive system: the dual-check filters samples safely, the uncertainty loss guides adaptation only on those safe samples, and the GMM confirms the separation before we let that adapted knowledge solidify into our prototypes.

Lu: It’s an elegant combination of probabilistic modeling for separation and calibrated uncertainty for learning, which tackles both major hurdles in open-set TTA simultaneously. I think this is a very solid methodological foundation because it tackles two distinct problems at once.

Meng: If we look at the results mentioned, the paper shows state-of-the-art performance on CIFAR10/one hundred and TinyImageNet with accuracy boosts over existing methods, which validates that this complex framework actually yields tangible improvements in recognition and OOD detection metrics like AUROC.

Title and authors: Lalam: I'm really excited because these results show that we can maintain high known-class accuracy while simultaneously achieving superior Out-of-Distribution detection metrics, which is exactly what we need for safety in real-world applications.

Tom: So, to wrap up the discussion on "ProtoDCS: Towards Robust and Efficient Open-Set Test-Time Adaptation for Vision-Language Models," it seems the authors have successfully created a framework that moves beyond brittle methods by using probabilistic separation and uncertainty modeling, all while keeping computational demands low through prototype-level updates.

Jane: It's a significant piece of research because it directly addresses the safety concerns associated with deploying VLMs in open-set environments where unknown data is inevitable. We’ve seen how this work allows for more reliable adaptation by separating what we know from what we don't, and using uncertainty to guide our learning process.

Lu: The implication here is that we can start thinking about deployment scenarios where the test stream isn't perfectly controlled, like in autonomous systems or industrial inspection, because the model has a much better mechanism for handling those shifts robustly.

Meng: Practically speaking, if this works as claimed across various backbones and complex datasets like TinyImageNet-C, it significantly lowers the barrier for deploying sophisticated vision models in scenarios that aren't pristine lab settings.

Lalam: For us, this means we can build more trustworthy AI systems that don't just perform well on clean data but actually maintain their reliability when faced with the messy reality of the real world.

Tom: Absolutely. So, as we wrap up our discussion on ProtoDCS: Towards Robust and Efficient Open-Set Test-Time Adaptation for Vision-Language Models, it’s clear this work offers a much more principled way to handle distribution shifts in open-set scenarios compared to previous hard thresholding techniques.

Jane: It really shows how integrating uncertainty modeling directly into the adaptation loop can lead to better calibrated predictions and a safer learning process for large vision models.

Lu: We should keep an eye on how this prototype-based separation mechanism could be adapted for other types of multimodal shifts, perhaps even text-to-image scenarios, given the general framework they’ve established here.

Meng: I'll be watching the implementation details closely to see if that prototype update mechanism can truly scale up efficiently for our production pipelines.

Lalam: I think this paper provides a very clear roadmap for building more reliable and adaptable vision systems that operate safely outside of perfectly controlled training environments.

The paper's summary: Tom: So, to recap, this paper introduces Prototype-based Double-Check Separation, or ProtoDCS, which is a new way to handle vision-language models when they encounter data they haven't seen during training in the real world.

Jane: Exactly, Tom; it’s basically a robust framework that uses a careful sequence of checks and updates to tell the difference between data that's shifted but still familiar and completely new stuff.

Lu: I think what really stands out about this summary is how they systematically tackle the core issue of separating those samples without just relying on simple, brittle cut-offs. It’s not just one check; it’s a whole verification pipeline built around a Gaussian Mixture Model to give us a probability of whether something is in or out of distribution.

Meng: From an engineering standpoint, that probabilistic modeling sounds heavy, but if it actually works to prevent the model from getting confused by bad data during adaptation, that’s valuable. I'm interested in how they manage the computational load when they are running this whole check sequence on a large VLM.

Lalam: For me, what's super exciting is how this approach ensures we don't just adapt blindly; it uses uncertainty to guide the learning process, which means the resulting model is better calibrated and safer when it makes decisions in ambiguous situations. That kind of reliability could really change how we deploy these powerful vision systems.

Tom: Right, Lalam, that calibration aspect is huge because it moves us away from just hoping for the best during deployment. Jane, can you explain what they mean by using uncertainty to guide adaptation specifically?

Jane: Well, they use a loss function called Luncertainty that models both the inherent noise in the data and how much the model itself doesn't know about a sample, which stops us from overconfident updates. It ensures we only adjust our prototypes when we have high confidence in what we're seeing.

Lu: That uncertainty modeling ties back into their separation mechanism; it’s not just about sorting; it’s about making sure the adaptation steps taken are actually informed by reliable data, which is a really deep way to handle distribution shifts. I wonder if this separation principle can be generalized beyond just vision and language tasks.

Meng: I'm still focused on the practical application of that prototype update; they keep the main VLM backbone frozen and only tweak these small residual matrices, which means we avoid retraining everything every time we get new data for adaptation. That keeps things running in real-time, which is what I need to see to make this useful outside of a research paper.

Lalam: And that asymmetry between updating text prototypes slowly through a moving average versus visual prototypes based on high-fidelity samples is really smart for maintaining semantic stability while allowing the visuals to adapt quickly. It’s a nice balance.

Tom: So, we're looking at a framework that combines probabilistic sample filtering with uncertainty-aware learning and efficient prototype updates, all working together to make VLMs much more reliable in open-set environments. Jane, what do you think this means for the future of deploying these kinds of models?

Jane: It means we can finally trust these models more when they encounter novel situations because we have a clear mechanism to distinguish between known shifts and truly new data, leading to systems that are much more dependable in complex settings.

Lu: I’m really optimistic about this; this methodology gives us a solid foundation for building multimodal AI that doesn't just work on clean training sets but can actually function reliably in the messy reality of the world. We might see this separation principle applied across different modalities soon.

Meng: For me, it means we have a clearer path toward deploying these kinds of models on edge devices where computational power is limited, because they aren't bogged down by massive backpropagation steps during adaptation. That’s a huge win for deployment feasibility.

Lalam: It really feels like this work pushes us toward building AI that's not just smart in the lab, but genuinely helpful and safe when it interacts with people and environments outside of those perfect conditions. That’s the kind of impact we want to see most from this research.

The paper's improvements: Tom: So, to wrap up on the improvements section, we're looking at three major innovations in ProtoDCS: a better separation mechanism, an uncertainty-aware loss function for adaptation, and a lightweight way to update prototypes without retraining the whole model.

Jane: That’s right; it’s like giving the AI a smarter set of tools instead of just using one simple method for every problem, which makes the whole process much more robust and safer.

Lu: The separation mechanism replacing brittle hard thresholds with GMM verification is really clever because it handles those tricky boundary cases where data might be ambiguous, preventing noisy gradients from corrupting the core knowledge base. It’s a sophisticated way to filter information before we even let it influence the model.

Meng: And that lightweight prototype update strategy is what gets my attention; keeping the main VLM backbone frozen means we can adapt quickly and efficiently on edge hardware without needing massive computational resources, which is crucial for real-world deployment scenarios.

Lalam: From a cultural standpoint, this kind of reliable adaptation means we can deploy AI in more sensitive areas because the system is calibrated to know when it’s unsure, leading to systems that are far more trustworthy in complex interactions. That level of safety builds public confidence in how we use these tools.

Tom: Exactly, Lalam; those improvements show a clear path toward building VLMs that aren't just accurate on clean data but are also resilient and safe when they run into real-world unpredictability. Jane, can you explain the difference between the uncertainty-aware loss and the old entropy minimization we discussed?

Jane: The uncertainty-aware loss explicitly captures both aleatoric and epistemic uncertainty, which means it tells the model whether its confusion is because the input data is inherently noisy or because it simply lacks sufficient knowledge about that specific visual scene. It’s much more informative than just minimizing entropy.

Lu: That distinction between those two types of uncertainty is vital; it allows us to tailor our adaptation strategy based on the source of the uncertainty, ensuring we only update the model when necessary and in a well-informed way. I think this detailed modeling of uncertainty is what really elevates this framework beyond standard TTA methods.

Meng: It makes sense that distinguishing between data noise and model ignorance helps us manage resources better; we don't waste time updating prototypes based on random noise in the input stream. That focus on high-fidelity samples being the drivers for adaptation is a very practical design choice.

Lalam: When you combine that with the asymmetric update strategy—updating text knowledge steadily but only adapting visuals selectively—it creates a much more balanced and stable AI system that preserves core understanding while letting it evolve visually where it needs to. That balance is essential for long-term, successful integration of AI into our infrastructure.

Tom: So, we’ve seen how these specific technical improvements—probabilistic separation, uncertainty loss, and selective updates—work together to create a much more robust and efficient adaptive system for vision-language models. Jane, what's the big picture implication of this whole approach for the world?

Jane: The implication is that we can start deploying AI in situations where perfect training data isn't available, making our technology far more adaptable and less fragile in real-world environments. This opens up possibilities for robotics and inspection tools that can handle unexpected shifts without breaking down.

Lu: I see the bigger picture here as a foundation for truly generalizable multimodal AI; if we can solve this separation problem robustly, we pave the way for systems that can reason across visual and textual domains in unpredictable settings. That is a massive area of creative potential.

Meng: On the practical side, this means less time spent on tedious retraining cycles and more time focusing on developing applications that actually solve complex problems in industry, because the AI itself is becoming much more reliable under messy conditions.

Lalam: For us, it’s about moving toward a culture where we design AI systems not just for perfect scenarios but for the real world, making our tools indispensable and genuinely useful in challenging circumstances.

Conclusion: Tom: So, to wrap up on "ProtoDCS: Towards Robust and Efficient Open-Set Test-Time Adaptation for Vision-Language Models," we've seen how this framework uses a double-check separation mechanism combined with uncertainty modeling to make adaptation much more reliable in open-set scenarios.

Jane: It really shows how careful, step-by-step verification can lead to much safer and more calibrated AI systems when they encounter data they haven't seen before.

Lu: I think the main contribution here is providing a mathematically sound way to decouple the identification of distribution shifts from the actual adaptation process, which is a really high-level concept for multimodal learning.

Meng: The practical implication for us is that we can start thinking about deploying these VLMs in more unpredictable industrial settings because we have a solid methodology to ensure they don't just guess when they see something new.

Lalam: For me, this research means our AI becomes much more trustworthy in complex human-AI interactions because it understands its own limitations and uncertainty better than before. That kind of transparency is what will truly improve how people feel about using advanced AI tools.

Tom: Exactly, Lalam; that focus on reliability and safety through structured verification is what sets this paper apart from previous TTA methods. Jane, do you have any final thoughts on the overall impact?

Jane: I think the idea of integrating uncertainty modeling directly into the adaptation loop is a big step forward for making AI predictions truly honest about their confidence levels.

Lu: It opens up a lot of creative avenues for how we structure future multimodal learning tasks, moving beyond simple classification to more nuanced reasoning under uncertainty.

Meng: From my side, the efficiency gains in prototype-level updates mean we can achieve this level of robustness on much smaller and less powerful hardware than previously thought possible.

Lalam: I just feel like this work reinforces a culture of cautious and thoughtful deployment, where we prioritize safety and calibration alongside raw performance metrics.

1South China Agricultural University · Pazhou Laboratory · South China University of Technology

cs.CV, cs.AI

Submitted: 2026-02-27

Updated: 2026-10-01

Code: https://github.com/O-YangF/ProtoDCS

Importance score: 92/100

The gist: Large-scale Vision-Language Models (VLMs) face significant challenges in real-world deployment due to distribution shifts, and existing Test-Time Adaptation (TTA) methods fail in open-set scenarios

Key concepts

Double-Check Separation: This mechanism uses two thresholds to initially filter samples. A strict threshold selects confident samples for a cache, while a looser one identifies trustworthy samples. This initial check helps segregate potential in-distribution and out-of-distribution data before adaptation begins.
Uncertainty-Aware Loss (Luncertainty): Instead of just minimizing prediction error, this loss function explicitly models both aleatoric (inherent noise) and epistemic (model uncertainty) uncertainty. This prevents the model from becoming overconfident by ensuring safe updates based on better-calibrated predictions in open-set scenarios.
Gaussian Mixture Model (GMM) Verification: A GMM is used to probabilistically separate samples based on their openness scores. It models the distribution of these scores as arising from two sources: in-distribution and out-of-distribution data. This provides a robust way to decide if a sample is truly in-distribution or not.
Prototype-Level Update Framework: This lightweight update strategy modifies learnable prototypes instead of the entire VLM backbone. By keeping the main model frozen, it saves computation and allows for selective updates, where textual prototypes use a stable moving average and visual ones are updated based on high-fidelity samples.

Terminology

Adaptation — 161× in this paper · explained in 1 episode
Verification — 46× in this paper · explained in 2 episodes
Accuracy — 33× in this paper · explained in 1 episode
Entropy Minimization — 32× in this paper
Distribution Shift — 22× in this paper · explained in 5 episodes
Robustness — 16× in this paper · explained in 5 episodes
Core — 14× in this paper · explained in 2 episodes
AUC — 11× in this paper · explained in 2 episodes
Alignment — 11× in this paper · explained in 1 episode
Calibration — 10× in this paper · explained in 7 episodes
Gaussian Mixture Model — 10× in this paper
Open-Set Test-Time Adaptation — 10× in this paper
Epistemic Uncertainty — 9× in this paper · explained in 9 episodes
Aleatoric Uncertainty — 7× in this paper · explained in 4 episodes
Gaussian Mixture Model (GMM) — 7× in this paper · explained in 2 episodes
Consistency — 6× in this paper · explained in 3 episodes
Fidelity — 6× in this paper · explained in 2 episodes
Ablation — 5× in this paper · explained in 1 episode
Evidence-Driven Uncertainty-Aware — 5× in this paper
Generalization — 5× in this paper · explained in 13 episodes
Loss Function — 5× in this paper · explained in 2 episodes
Vision-Language Models (VLMs) — 5× in this paper · explained in 6 episodes
Benchmark — 4× in this paper · explained in 2 episodes
Computational overhead — 4× in this paper · explained in 2 episodes
Cumulative Moving Average — 4× in this paper
Domain Shift — 4× in this paper · explained in 4 episodes
Online Adaptation — 4× in this paper · explained in 3 episodes
Test-time Adaptation (TTA) — 4× in this paper · explained in 3 episodes
Ablation Studies — 3× in this paper · explained in 1 episode
Aleatoric Uncertainty (AU) — 3× in this paper · explained in 1 episode
Cosine Similarity — 3× in this paper · explained in 3 episodes
Domain Adaptation — 3× in this paper · explained in 5 episodes
Edge Devices — 3× in this paper · explained in 2 episodes
Redundancy — 3× in this paper · explained in 2 episodes
Regularization — 3× in this paper · explained in 3 episodes
Closed-Loop System — 2× in this paper · explained in 4 episodes
Multimodal Learning — 2× in this paper · explained in 2 episodes
Pruning — 2× in this paper · explained in 4 episodes
Representation Learning — 2× in this paper · explained in 2 episodes
Semantic Drift — 2× in this paper · explained in 3 episodes
AdamW Optimizer — 1× in this paper · explained in 1 episode
Adversarial Examples — 1× in this paper · explained in 2 episodes
Aggregation — 1× in this paper · explained in 1 episode
Benchmarking — 1× in this paper · explained in 3 episodes
Clustering — 1× in this paper · explained in 2 episodes
Composite Loss Function — 1× in this paper · explained in 2 episodes
Contrastive Language-Image Pre-Training — 1× in this paper
Covariate Shift — 1× in this paper · explained in 2 episodes
Data Augmentation — 1× in this paper · explained in 7 episodes
Diffusion Model — 1× in this paper · explained in 8 episodes
Downstream Tasks — 1× in this paper · explained in 2 episodes
Exponential Moving Average — 1× in this paper
Exponential Moving Average (EMA) — 1× in this paper · explained in 2 episodes
Fisher Information — 1× in this paper · explained in 3 episodes
Foundation Model — 1× in this paper · explained in 3 episodes
Foundation Models — 1× in this paper · explained in 7 episodes
Generalizability — 1× in this paper · explained in 6 episodes
Grounding — 1× in this paper · explained in 3 episodes
Model-Agnostic — 1× in this paper · explained in 2 episodes
Multimodal Fusion — 1× in this paper · explained in 3 episodes
Pseudo-labeling — 1× in this paper · explained in 3 episodes
Uncertainty Quantification — 1× in this paper · explained in 8 episodes
While Test-Time Adaptation — 1× in this paper
Zero-Shot Generalization — 1× in this paper · explained in 5 episodes

Summary

Large-scale Vision-Language Models (VLMs) face significant challenges in real-world deployment due to distribution shifts, and existing Test-Time Adaptation (TTA) methods fail in open-set scenarios where test streams contain both covariate-shifted in-distribution (csID) and out-of-distribution (csOOD) data. This work proposes Prototype-based Double-Check Separation (ProtoDCS), a robust framework that effectively separates csID and csOOD samples while enabling safe and efficient adaptation of VLMs to csID data.

The gist

ProtoDCS introduces a novel framework for open-set TTA that employs a probabilistic double-check separation mechanism with Gaussian Mixture Model (GMM) verification to robustly handle distribution shifts, coupled with an evidence-driven adaptation strategy utilizing uncertainty-aware loss and efficient prototype-level updates, mitigating overconfidence and reducing computational overhead.

How it works

ProtoDCS systematically resolves the difficulties in existing open-set TTA methods through a cohesive design centered on reliable separation and efficient adaptation. The framework operates in four sequential stages for an incoming test sample:

The First-Check stage computes an initial openness score, denoted as Sopen(x), to segregate samples using dual-threshold filtering. A strict threshold, Θa (e.g., the 30th percentile), populates a diversity-aware visual cache (Cv) with confident samples, while a looser threshold, Θb (e.g., the 60th percentile), identifies trustworthy samples (xtrust) that warrant further assessment.
The Evidence-driven Temporary Prototype Generation stage performs safe adaptation exclusively on trustworthy samples. This involves updating learnable residual matrices (∆Pt and ∆Pv) through gradient descent to generate temporary prototypes, P′t and P′v, while keeping the VLM backbone frozen. The optimization objective minimizes a composite loss: min L = Luncertainty + λalignLalign, where Luncertainty explicitly models both aleatoric uncertainty (AU) and epistemic uncertainty (EU), replacing entropy minimization to prevent overconfidence.
The Final-Verification stage provides definitive csID/csOOD separation using a Gaussian Mixture Model (GMM). The GMM models the openness scores as arising from two underlying distributions, allowing for probabilistic separation of csID and csOOD samples based on their mean openness score, which is mapped to the OOD component. This decision is made by checking P(ccsOODSopen(x)) against a threshold Θp (e.g., 0.5). Only samples confirmed as csID contribute to the final prototype evolution, ensuring adaptation is driven exclusively by high-fidelity csID data.

Key Innovations and Mechanisms

The paper introduces three core innovations to address the limitations of existing methods:

A novel double-check separation mechanism employing probabilistic Gaussian Mixture Model (GMM) verification to replace brittle thresholding. This mechanism handles ambiguous boundary cases robustly by modeling the openness score distribution, ensuring that erroneous gradients from misclassified csOOD data corrupt the representations of known classes are avoided.
An evidence-driven adaptation strategy utilizing an uncertainty-aware loss (Luncertainty) to directly counter the overconfidence issue of entropy minimization. This loss explicitly models both aleatoric and epistemic uncertainty, ensuring safe model updates through better-calibrated predictions in open-set scenarios.
A lightweight prototype-level update framework that keeps the VLM backbone frozen, effectively overcoming the computational bottleneck of gradient backpropagation through the entire backbone. Prototype evolution is asymmetric: textual prototypes are updated via a cumulative moving average (CMA) for stability, while visual prototypes undergo more selective updates based on high-fidelity samples confirmed by both verification stages.

Experimental Validation

Extensive experiments on CIFAR10/100-C and Tiny-ImageNet-C demonstrate that ProtoDCS achieves state-of-the-art performance, significantly boosting both known-class accuracy (Acc) and OOD detection metrics (AUROC). The results show that ProtoDCS outperforms all baselines across every metric. Specifically, it achieves 95.59% and 24.03% on AUROC and FPR@TPR95 on CIFAR-10-C, showing a relative improvement over DPE by 10.69% and 58.15% for OSCR on CIFAR-10-C. Furthermore, the framework proves robust to hyperparameter selection, with its accuracy remaining stable across various threshold settings and window sizes. The analysis of the cold-start adaptation dynamics confirms that the GMM activation point at 100 samples initiates a steep and steady ascent in accuracy, validating the effectiveness of the progressive double-check mechanism in handling distribution shifts.

Improvements for AI systems

Here are specific improvements for AI systems based on the ProtoDCS framework, detailing what these improved systems can achieve:

) Robust Open-Set Test-Time Adaptation (OSTTA) for Vision-Language Models (VLMs): Instead of relying on brittle hard thresholds or blind entropy minimization, the system will utilize the Prototype-based Double-Check Separation mechanism.

A novel two-stage verification process will be implemented: a First-Check stage using dual dynamic thresholds to filter samples into a diversity-aware visual cache, followed by a Final Verification stage employing Gaussian Mixture Model (GMM) verification on openness scores to probabilistically separate covariate-shifted in-distribution (csID) from out-of-distribution (csOOD) data.
The adaptation strategy will employ an Evidence-Driven Uncertainty-Aware (EDUA) loss function, which explicitly models both aleatoric and epistemic uncertainty, replacing entropy minimization. This prevents overconfident predictions on noisy or boundary samples and ensures that only high-fidelity csID samples drive prototype evolution.

) Enhanced Model Robustness in Dynamic Real-World Environments: The improved system can operate safely in open-set scenarios (e.g., autonomous driving, industrial inspection) where the test stream contains both known shifted data and novel objects. It will achieve this by maintaining high known-class accuracy while simultaneously achieving superior Out-of-Distribution (OOD) detection metrics (AUROC, FPR@TPR95), ensuring it correctly flags potential risks without corrupting its internal representations.

) Computationally Efficient Online Adaptation for Large Models: The system will maintain a frozen VLM backbone and perform all adaptation exclusively at the prototype level using evidence-driven temporary prototype optimization via residual updates. This eliminates the prohibitive computational cost of gradient backpropagation through the entire backbone, enabling safe and efficient adaptation in real-time or low-latency deployment on edge devices (e.g., mobile sensors, embedded systems).

) Safe and Calibrated Uncertainty Estimation: By integrating the EDUA loss, the system will produce better-calibrated uncertainty estimates for predictions. It can distinguish between true model ignorance (epistemic uncertainty) and inherent data noise (aleatoric uncertainty), leading to more reliable decision-making in ambiguous situations, thus preventing negative adaptation where OOD samples are misclassified as known classes.

) Modality-Specific Adaptation: The system will utilize an asymmetric prototype update strategy. Textual prototypes will be updated via a stable Cumulative Moving Average (CMA), preserving semantic knowledge, while visual prototypes undergo selective updates based on high-fidelity, verified csID samples. This allows the model to rapidly adapt its visual features to domain shifts while maintaining the semantic stability of its language understanding.

) State-of-the-Art Performance Across Architectures and Complex Scenarios: The improved system is architecturally general, achieving state-of-the-art performance on standard benchmarks (CIFAR/ImageNet) and demonstrating superior scalability on complex datasets (TinyImageNet), showing consistent improvements over existing VLM TTA methods across various backbone architectures.