Generalist-Specialist Mixture-of-Experts for Rare Pathology Detection in Multimodal Imaging

arXiv:2609.18688 · cs.CV, cs.AI · Submitted 2026-09-16 · Read on arXiv

cs.CV, cs.AI

Submitted: 2026-09-16

Updated: 2026-09-16

License: http://creativecommons.org/licenses/by/4.0/

The gist: AI models for multimodal medical imaging must balance modality-specific specialization with cross-modal shared representations, a trade-off that pure Mixture-of-Experts (MoE) architectures currently

Terminology

Abstract

AI models for multimodal medical imaging must balance modality-specific specialization with cross-modal shared representations, a trade-off that pure Mixture-of-Experts (MoE) architectures currently fail to satisfy. Expert-based routing improves in-domain learning but may sacrifice cross-modal signals, which appear particularly important for rare (low-prevalence) pathologies in our experiments. To resolve this, we introduce Generalist-Specialist-MoE (GS-MoE), a two-branch (MoE) architecture that couples a cross-modal generalist model with distinct modality-specific specialists (experts) via domain-constrained feature fusion. On RadImageNet (1.35M images, 165 pathologies, three modalities), GS-MoE recovers detection of six low-prevalence pathologies on which every baseline scores F1 = 0, with per-class gains up to +0.60 F1. It attains this while even slightly exceeding dense and specialist-only MoE aggregate baselines (MCC 0.770), while using about 53% fewer active parameters at inference than the strongest investigated dense model.

Sources

Related papers