How Medical VLMs Underutilize Their Vision Encoders: A Dermatology Perspective

arXiv:2609.36557 · cs.CV, cs.AI · Submitted 2026-09-29 · Read on arXiv

cs.CV, cs.AI

Submitted: 2026-09-29

Updated: 2026-09-29

Terminology

• Accuracy — 38× in this paper · explained in 1 episode
• Linear probe — 30× in this paper · explained in 2 episodes
• Fine-Tuning — 20× in this paper · explained in 5 episodes
• Direct Answer — 18× in this paper
• Adaptation — 12× in this paper · explained in 1 episode
• Basal Cell Carcinoma — 11× in this paper
• Lupus Erythematosus — 11× in this paper
• Vision Attention Ratio — 11× in this paper
• Ablation — 7× in this paper · explained in 1 episode
• Foundation Model — 7× in this paper · explained in 3 episodes
• Grounding — 7× in this paper · explained in 3 episodes
• Linear Probing — 7× in this paper · explained in 2 episodes
• Ablation study — 6× in this paper · explained in 1 episode
• Attention Entropy — 6× in this paper · explained in 2 episodes
• Abstention — 5× in this paper · explained in 1 episode
• Squamous Cell Carcinoma — 5× in this paper
• Visual Grounding — 5× in this paper · explained in 2 episodes
• Foundation Models — 3× in this paper · explained in 7 episodes
• LoRA Fine-Tuning — 3× in this paper · explained in 3 episodes
• Vision-language models — 3× in this paper · explained in 2 episodes
• alpha — 3× in this paper · explained in 1 episode
• AdamW Optimizer — 2× in this paper · explained in 1 episode
• Alignment — 2× in this paper · explained in 1 episode
• Benchmark — 2× in this paper · explained in 2 episodes
• Consistency — 2× in this paper · explained in 3 episodes
• Transformer — 2× in this paper · explained in 2 episodes
• Vision-Language Models (VLMs) — 2× in this paper · explained in 6 episodes
• Adaptability — 1× in this paper · explained in 1 episode
• Catastrophic Forgetting — 1× in this paper · explained in 5 episodes
• Core — 1× in this paper · explained in 2 episodes
• Domain Adaptation — 1× in this paper · explained in 5 episodes
• Feature Extraction — 1× in this paper · explained in 2 episodes
• Generalizability — 1× in this paper · explained in 6 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Hallucination — 1× in this paper · explained in 4 episodes
• In-Context Learning — 1× in this paper · explained in 4 episodes
• LoRA Adapters — 1× in this paper · explained in 3 episodes
• Low-Rank Adaptation (LoRA) — 1× in this paper · explained in 9 episodes
• Ontology — 1× in this paper · explained in 2 episodes
• Optimal Transport — 1× in this paper · explained in 5 episodes
• Robustness — 1× in this paper · explained in 5 episodes
• Steering — 1× in this paper · explained in 2 episodes
• VLM Zero-Shot Fine-Tuning — 1× in this paper

Sources

• MedGemma Technical Report
• Gemma 3 Technical Report
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• eSkinHealth: A Multimodal Dataset for Neglected Tropical Skin Diseases
• Doctor Approved: Generating Medically Accurate Skin Disease Images through AI-Expert Feedback
• InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
• Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization
• Derm1M: A Million-scale Vision-Language Dataset Aligned with Clinical Ontology Knowledge for Dermatology
• Look-Back: Implicit Visual Re-focusing in MLLM Reasoning
• MM-Skin: Enhancing Dermatology Vision-Language Model with an Image-Text Dataset Derived from Textbooks
• Investigating the Catastrophic Forgetting in Multimodal Large Language Models
• BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs
• Adapting in the Dark: Efficient and Stable Test-Time Adaptation for Black-Box Models
• InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models

Related papers

• Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
• AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
• Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
• MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
• TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
• A Survey on Efficient Vision-Language-Action Models