DuetMoE: Coupling Inter- and Intra-Subgroup Robustness for Fair Medical Image Analysis

Medical image segmentation models often perform unevenly across different patient subgroups, and existing fairness methods frequently fail by treating each subgroup as internally homogeneous, which obscures difficult cases within those groups. This study proposes DuetFair, a dual-axis fairness framework implemented via FairDRO, to jointly address inter-subgroup adaptation and intra-subgroup robustness in medical image segmentation. The proposed mechanism aims to ensure that model outputs are reliable for individual patients across various demographic and clinical subgroups by accounting for both distribution shifts between groups and hard samples within those groups.

The gist

FairDRO combines subgroup-aware distributionally robust optimization (DRO) loss aggregation with a subgroup-conditioned mixture-of-experts (dMoE) representation to simultaneously capture inter-subgroup heterogeneity and intra-subgroup variation in medical image segmentation.

How it works: The DuetFair Mechanism

The DuetFair mechanism views subgroup fairness as a joint problem of inter-group adaptation and intra-group robustness. This is realized through two complementary axes:

1. Inter-Subgroup Axis (Representation Learning): FairDRO uses dMoE to address inter-subgroup variation at the representation level. Specifically, it adapts intermediate features using a subgroup-conditioned mixture of expert transformations defined by Equation (2), which allows different experts to capture heterogeneous demographic and clinical patterns. This places inter-subgroup adaptation in the representation module, rather than relying only on loss weights to capture the group-wise differences.

2. Intra-Subgroup Axis (Robust Loss Aggregation): The intra-subgroup axis is handled through a distributionally robust view of subgroup risk. Instead of evaluating loss under a uniform empirical mean, FairDRO defines a subgroup-specific ambiguity set Ug(Pbg) containing distributions close to Pbg. The robust risk within subgroup g is defined as the supremum over this set:

Rrob g(θ, ϕ):= sup Qg∈Ug(Pbg) E(x,y)∼Qg lf dMoE θ,ϕ (x, g), y. (3)

This formulation makes the robust risk sensitive to high-loss samples within subgroup g, ensuring that hard individual cases are prioritized relative to their own subgroup while the subgroup structure remains explicit.

How it works: The FairDRO Objective

The final objective couples these two axes through a weighted aggregation of the robust risks. The overall objective is formulated as:

min θ,ϕ LFairDRO(θ, ϕ) = X g∈G wgRrob g(θ, ϕ), (4)

Here, Rrob g(θ, ϕ) is the within-subgroup robust risk evaluated using the dMoE predictor. The weight wg aggregates subgroup-level robust risks. In the implementation described in Equation (4), we set wg = 1/G, so FairDRO uniformly aggregates subgroup risks without manually upweighting any subgroup. This design ensures that subgroup information guides dMoE routing, while robustness is imposed through the worst-case distribution over samples within each subgroup.

Evaluation and Results

FairDRO was evaluated on three medical image segmentation benchmarks: Harvard-FairSeg, HAM10000, and an in-house 3D radiotherapy target cohort. The results demonstrate that FairDRO achieves the best equity-scaled performance on Harvard-FairSeg and improves worst-case subgroup performance on HAM10000 under both age- and race-based grouping schemes. On the 3D radiotherapy target cohort, FairDRO further improves worst-group Dice by 3.5 points (↑ 6.0%) under the tumor-stage grouping and by 4.1 points (↑ 7.4%) under the institution grouping over the strongest baseline. Qualitative analysis shows that FairDRO produces tighter per-patient prediction across both minor and major groups, rather than a uniform shift toward a subgroup-mean template.

Limitations and Future Directions

The paper notes two primary limitations:

1. FairDRO may not always produce the largest gain when predefined subgroup labels already explain most of the performance gap, suggesting its value is maximized when subgroup labels only partially describe patient difficulty.

2. FairDRO relies on subgroup attributes (e.g., race, age, tumor T-stage) during training and inference; future work plans to extend DuetFair on the attribute-aware side through continuous conditioning and on the attribute-agnostic side by developing methods such as latent-attribute inference, attribute imputation, and self-supervised subgroup discovery. Additionally, an alternative objective variant, FairDRO-Penalty (Equation 16), was explored but showed that direct coupling of robustness with the objective leads to more stable improvements than simply appending a DRO term to the empirical loss.

As a fastidious researcher, I have analyzed the proposed DuetFair mechanism and FairDRO framework. The core innovation lies in jointly addressing inter-subgroup heterogeneity (via subgroup-conditioned distributionally robust optimization, dMoE) and intra-subgroup hidden failures (via the distributionally robust loss term).

Here are the specific improvements and capabilities that can be derived from this research, targeted at enhancing medical image segmentation systems:

The proposed system, FairDRO, is a dual-axis fairness framework that integrates subgroup-aware representation learning with within-subgroup robustness. It fundamentally shifts the training objective from simple average performance to a mechanism that prioritizes high-loss samples within each subgroup while adapting representations across different subgroups.

Here are the specific improvements and capabilities this system enables:

1. The ability to mitigate intra-group hidden failures.

2. The capability to achieve superior worst-case subgroup performance under heterogeneous distributions.

3. Enhanced reliability for minority or difficult patient subgroups without sacrificing overall population average accuracy significantly (as demonstrated in Table 6).

Specific, detailed improvements and resulting capabilities:

1. An AI system can be trained to perform medical image segmentation (e.g., tumor delineation, organ boundary tracing) where the performance is not uniformly good across different demographic or clinical subgroups (e.g., race, age groups, or institutional settings).

2. The system will explicitly learn subgroup-aware feature representations using a Distributionally Robust Mixture-of-Experts (dMoE). This means the model can dynamically route input features to the most appropriate expert based on the patient's subgroup attribute (e.g., routing an image from a specific age group to an expert trained predominantly on similar anatomical variations).

3. The system will incorporate a distributionally robust loss aggregation that acts as an intra-subgroup robustness mechanism. This mechanism prevents high-loss, difficult samples within a specific subgroup (which would otherwise be washed out by the subgroup average) from being ignored during training. The model is forced to learn representations that are stable and accurate for these hard cases, even when they represent a small subset of the total data.

4. The resulting segmentation output will exhibit tighter per-patient prediction consistency across subgroups rather than just shifting toward a generalized subgroup mean. This means the system produces highly reliable results for individual patients, reducing the risk associated with misdiagnosis in vulnerable patient populations (e.g., ensuring accurate radiotherapy target contouring for rare T1 or T4 tumors, or precise optic cup delineation for specific ethnic groups).

5. In clinical deployment, this system can be configured to maintain high average performance across the entire population while simultaneously guaranteeing that the performance degradation experienced by any single subgroup is minimized (as measured by Equity-Scaled metrics like ES-Dice and ES-IoU).

This improved AI system moves beyond simple group balancing or loss penalization. It uses a sophisticated, two-pronged approach—adapting what the model sees (representation learning) and adapting how the model learns from hard examples (robust loss aggregation)—to ensure that no difficult patient case is hidden by the statistical average of their subgroup.

As medical AI expands across diverse healthcare settings worldwide, equitable performance across patient populations is becoming essential to trustworthy clinical use. Fairness in medical image analysis is often evaluated through average performance across predefined subgroups, yet similar subgroup averages can conceal substantial variation among individual patients. Therefore, a reliable medical AI requires addressing two complementary objectives: inter-subgroup fairness, which reduces performance disparities across groups, and intra-subgroup robustness, which protects poorly served patients within each group. To jointly address these objectives, we propose DuetMoE, a subgroup-aware mixture-of-experts framework that couples group-level adaptation with patient-specific clinical guidance, enabling more reliable medical image analysis for individual patients. For settings without linked clinical records, we further introduce DuetMoE+, which retains subgroup-routed experts and incorporates a KL-constrained distributionally robust objective to address intra-subgroup robustness. We evaluate our methods on PI-CAI, radiotherapy, and Harvard-FairSeg. Across four evaluation settings, our methods lead in overall or equity-scaled performance, reduce the inter-subgroup mean Dice gap by up to 30.7%, and raise intra-subgroup 25th-percentile Dice by up to 11.5 points over the strongest reported baselines.