UniPrompt-CL: Efficient Single-Inference Continual Learning for Medical Image Classification

Medical AI systems must remain reliable as data distributions shift across hospitals, imaging devices, and patient populations, yet privacy constraints often prevent storing or revisiting past patient data. We present UniPrompt-CL, a rehearsal-free domain-incremental learning framework for medical image classification using a single backbone and a single inference. Rather than relying on multiple backbones, repeated ViT inferences, or independent layer-wise prompt pools, UniPrompt-CL shares a unified prompt pool across transformer layers and expands it with only a few prompts per new domain. This reduces cross-layer redundancy while enabling efficient adaptation. On diabetic retinopathy, UniPrompt-CL improves average accuracy by 2.0 percentage points and substantially reduces forgetting relative to the strongest backbone-matched baseline. On skin cancer, it achieves competitive accuracy at the lowest training cost among DINOv2-B configurations. A controlled comparison varying only pool organization shows that the benefit arises from retention rather than learning each new domain: it is largest on the medical benchmarks, smaller on OfficeHome, and reverses on a heterogeneous six-stage DomainNet stream. These results suggest that cross-layer prompt sharing with lightweight expansion is practical for medical models requiring repeated retraining under limited on-site compute, while identifying the domain shifts under which it helps.