From Theory to Decision Rule: Calibrating the Noisy-Label Crossover for Vision-Language Model Weak Supervision Across Three Medical-Imaging Benchmarks

In medical imaging, small sets of expert labels are often combined with zero-shot pseudo-labels from a vision-language model (VLM). When the expert-labeled set is very small, the extra pseudo-labels reduce variance and improve accuracy. Once a classifier trained on expert labels alone reaches the accuracy of the weak labeler, however, adding pseudo-labels lowers test performance. We measure where this crossover occurs on PatchCamelyon histopathology, ISIC 2019 dermoscopy, and NIH ChestX-ray14 radiography using pseudo-labels from BiomedCLIP and PubMedCLIP across six student architectures (8 M to 89 M parameters). Weak labels stop helping after 20 -- 50 expert labels on ISIC 2019, about 100 on PatchCamelyon, and 250 -- 500 on NIH ChestX-ray14. Above those budgets, adding 10, 000 pseudo-labels reduces test macro-AUC by up to 0.14 points with BiomedCLIP (0.23 with PubMedCLIP) on ISIC 2019, 0.10 on PatchCamelyon, and 0.08 on NIH ChestX-ray14. Across the six student architectures, the weak-label ceiling stays within a 0.024 AUC range (0.841 -- 0.865), indicating that the VLM labeler, rather than student size, limits performance. Comparing VLM pseudo-labels with random label flips at the same error rate shows that systematic VLM mistakes yield 0.09 -- 0.20 lower AUC than random noise on PatchCamelyon and ISIC 2019, but 0.03 -- 0.04 higher AUC on 15-class NIH ChestX-ray14, where per-finding crossovers also span more than a 20 times range. A simple bias-variance rule estimated from the weak labeler's validation accuracy and small expert-only runs predicts whether pseudo-labels will help or hurt in all 30 statistically significant settings (39/42 overall).