Dual-Domain Cross-Modal Decoding for Clinical Text-Guided Medical Image Segmentation

Dual-Domain Cross-Modal Decoding (DD-CMD) is proposed for clinical text-guided pulmonary infection segmentation, integrating two complementary forms of language guidance during decoding. In the spatial domain, Text-Guided Spatial Cross-Attention (TGSA) aligns multi-scale visual tokens with text semantics and updates features through gated residual fusion. In the frequency domain, Spectral-Text Adaptive Modulation (STAM) applies a 2D DCT to compute learnable band-energy statistics and predicts text-conditioned FiLM parameters to recalibrate decoder channels for frequency-aware decoding. DD-CMD embeds TGSA and STAM into a coarse-to-fine decoder (7×7 → 56×56) and restores full-resolution masks using a lightweight two-stage refinement module. Experiments on QaTaCOV19 and MosMedData+ show that DD-CMD achieves 91.46% Dice / 84.26% mIoU and 81.95% Dice / 69.42% mIoU, respectively, with average gains of +1.96 Dice and +2.67 mIoU over the strongest prior baselines.

The key idea is to integrate two complementary forms of language guidance during decoding: spatial alignment indicates where the model should focus, while spectral calibration determines how decoder channels respond to frequency components that govern texture and boundary structure. Concretely, DD-CMD encodes images using ConvNeXt-Tiny and encodes clinical text using a frozen PubMedBERT. The model decodes masks through three progressive stages (7×7 → 14×14 → 28×28 → 56×56). At each stage, TGSA aligns visual tokens with text semantics and updates features through a gated residual fusion. In parallel, STAM applies a 2D discrete cosine transform (DCT), aggregates learnably gated band-energy statistics, and predicts text-conditioned feature-wise linear modulation (FiLM) parameters to recalibrate decoder channels in a frequency-aware manner. Finally, full 224×224 resolution is recovered through a two-stage refinement module that fuses shallow spatial features while preserving text consistency.

The contributions are three-fold: (1) a dual-domain cross-modal decoder integrating TGSA for text-guided spatial cross-attention with gated residual fusion, and STAM for frequency-aware channel recalibration using DCT band-energy statistics and FiLM conditioning; (2) a coarse-to-fine cross-modal design establishing image–text alignment through three decoder stages (7×7 → 56×56) and recovering full-resolution structure with a two-stage refinement module; (3) extensive experiments on QaTa-COV19 and MosMedData+, achieving 91.46% Dice / 84.26% mIoU on QaTa-COV19 and 81.95% Dice / 69.42% mIoU on MosMedData+, outperforming recent baselines.

The vision encoder is ConvNeXt-Tiny, producing a four-scale feature pyramid F56, F28, F14, F7 at spatial resolutions 56×56, 28×28, 14×14, 7×7 with channel dimensions 96, 192, 384, 768. The text encoder tokenizes clinical text, pads or truncates it to a maximum length of L=24, and encodes it with a frozen PubMedBERT (base-uncased) encoder, outputting token embeddings T ∈ RL×d with d=768. The token embeddings are used for cross-modal interaction, and mean pooling obtains a global descriptor t̄ ∈ Rd for stage-wise conditioning.

The decoder operates across three progressive stages: 7×7 → 14×14, 14×14 → 28×28, and 28×28 → 56×56. At each stage s, visual tokens V(s) ∈ RNs×Cs are processed by TGSA and STAM. TGSA first adds spatial inductive bias through a local convolutional mixer comprising depthwise 3×3 and pointwise 1×1 convolutions with BatchNorm and GELU. Then multi-head cross-attention is performed where locally-mixed visual tokens serve as queries Q and text embeddings serve as keys Kt and values Vt. An adaptive gate g(s) ∈ (0,1) scales text influence based on multi-modal agreement: g(s) = σ(MLP[Avg(Ṽ(s)); Avg(T(s))]), and Vatt = LN(V(s) + α(s) g(s) MHA(Q, Kt, Vt)), where α(s) is a learnable scale parameter and LN denotes LayerNorm. This is followed by a feed-forward network (FFN) and another LN layer.

STAM calibrates features in the frequency domain conditioned on text semantics. Given Xatt ∈ RCs×Hs×Ws (reshaped from attended tokens), channel-wise 2D DCT-II is applied to obtain frequency coefficients. A learnable gate G(s) ∈ (0,1)Hs×Ws softly weights informative frequency components when computing per-channel spectral energy: e(s)c = (1/(HsWs)) Σu,v σ(G(s)u,v) DCT(Xatt)c,u,v, for c = 1,..., Cs. This forms a compact spectral descriptor e(s) ∈ RCs. The spectral descriptor is concatenated with the global text vector t̄, and FiLM parameters (γ(s), β(s)) ∈ RCs are predicted using a two-layer MLP: [γ(s), β(s)] = MLP[e(s); t̄]. Channel-wise affine modulation is applied with tanh bounding for stability: Xcal = Xatt ⊗ (1 + tanh(γ(s))) + tanh(β(s)). At the end of each decoder stage, Xcal is upsampled by 2× via bilinear interpolation, concatenated with the corresponding higher-resolution encoder skip feature F(s), and refined using two 3×3 Conv-BN-GELU layers. Repeating this across three stages yields text-conditioned features at 56×56.

The high-resolution refinement module stops cross-modal decoding at 56×56 and recovers pixel-level details through two lightweight refinement blocks. Shallow image features are extracted directly from the input via small convolutional layers, producing S112 ∈ R48×112×112 (stride 2) and S224 ∈ R24×224×224 (stride 1). At each refinement resolution r ∈ 112, 224, incoming features are bilinearly upsampled, concatenated with the corresponding shallow skip Sr, and processed by two 3×3 Conv-BN-GELU layers. Lightweight FiLM conditioning from t̄ is applied to preserve text consistency at high resolution. Finally, a 1×1 convolution produces segmentation logits at 224×224, followed by a sigmoid to predict the final mask.

Experiments are conducted on two publicly available medical vision–language segmentation benchmarks: MosMedData+ and QaTa-COV19. MosMedData+ provides 2,729 COVID-19 lung CT slices with ground-truth infection masks and corresponding clinical text. QaTa-COV19 contains 9,258 chest X-ray images with lesion annotations and accompanying textual descriptions. The train/val/test splits are: MosMedData+ divided into 2,183/273/273 samples, and QaTa-COV19 split into 5,716/1,429/2,113 samples for training/validation/testing. The implementation uses PyTorch with PyTorch-Lightning on an NVIDIA RTX 3090Ti GPU (24 GB). All input images are resized to 224×224, with random zoom, horizontal/vertical flips, and rotation, followed by intensity normalization. The network is trained for 160 epochs with AdamW and cosine annealing (initial learning rate 5×10−5, ηmin = 10−6) using a batch size of 8. A combined Dice and cross-entropy objective is optimized. Dice and mIoU (%) are reported as primary metrics, with HD95 (pixels) additionally assessed for boundary quality.

Compared to state-of-the-art methods, DD-CMD achieves the best results on both datasets, with 91.46/84.26 Dice/mIoU on QaTa-COV19 and 81.95/69.42 on MosMedData+. Compared to MMI-UNet on QaTa-COV19, DD-CMD improves by +0.58 Dice and +0.98 mIoU. On MosMedData+, it surpasses the best prior Dice by +3.33 (MAdapter) and the best prior mIoU by +4.35 (RecLMIS), while using 47.77M parameters and 20.31 GFLOPs, comparable to several baselines.

Ablation studies analyze the contribution of each component. The image-only baseline performs the worst on both datasets, highlighting the benefit of clinical text. Keeping text but disabling both TGSA and STAM improves performance only slightly, indicating naive conditioning is insufficient. Enabling TGSA brings a clear jump by improving text–image alignment, and adding STAM further boosts overlap while reducing boundary error. The high-resolution refinement primarily sharpens boundaries, while FiLM-style parameters and local mixing in TGSA bring additional consistent gains. The full model is best and most stable across datasets.

Text length ablation varies L from 8 to 40. Very short prompts reduce overlap and boundary quality because truncation can remove lesion descriptors needed for localization and shape. Performance improves as L increases and then plateaus: L=24 provides a strong, stable default across datasets, while longer texts (L=32, 40) provide no consistent gains and can slightly degrade due to diminishing returns and added noise from extra tokens. L=24 is used in all main experiments.

Progressive feature visualization shows how the prediction improves as decoder and refinement blocks are added. Decoder (D1) gives a rough, low-resolution localization with broad activations and loose boundaries. Adding D2 and D3 strengthens lesion-specific responses and suppresses background, making the mask semantically reliable and better covering the infection extent. After this point, refinement mainly improves boundaries: R1 sharpens edges and reduces leakage into nearby tissue, and R2 closes small gaps and stabilizes thin structures.

The conclusion states that DD-CMD integrates spatial and spectral language guidance within a unified decoding framework. TGSA uses text-guided spatial cross-attention to align visual tokens with clinical semantics, whereas STAM calibrates decoder features through DCT band-energy statistics and text-conditioned FiLM parameters for frequency-aware decoding. This dual-domain design is implemented in a coarse-to-fine decoder (7×7 → 56×56), with a refinement pathway to recover full-resolution masks. Experiments on QaTa-COV19 and MosMedData+ demonstrate the effectiveness of DD-CMD with strong overlap and boundary accuracy.

Improvements to AI Systems:

1. Dual-Domain Cross-Modal Decoding (DD-CMD) Integration: Implement a decoder that simultaneously processes spatial and frequency domains for text-guided segmentation. This enables the AI to align visual features with language semantics (spatial) while recalibrating channel responses based on spectral band-energy statistics (frequency), improving texture and boundary detection in medical imaging.

2. Text-Guided Spatial Cross-Attention (TGSA) with Gated Residual Fusion: Add a mechanism where visual tokens serve as queries and text embeddings as keys/values, with an adaptive gate (derived from multi-modal agreement) controlling text influence. This allows the AI to dynamically focus on lesion-relevant regions based on clinical descriptions, reducing false positives and improving localization.

3. Spectral-Text Adaptive Modulation (STAM): Apply 2D DCT to compute learnable band-energy statistics per channel, then predict text-conditioned FiLM parameters (γ, β) to modulate features. This enables the AI to adaptively emphasize frequency components tied to infection texture and boundaries, improving segmentation accuracy on ambiguous or low-contrast regions.

4. Coarse-to-Fine Multi-Stage Decoding with Progressive Upsampling: Use three decoder stages (7×7 → 56×56) that integrate TGSA and STAM at each resolution, followed by a two-stage refinement module (56×56 → 224×224). This allows the AI to first establish global semantic alignment, then progressively refine local details, balancing context and precision.

5. Lightweight High-Resolution Refinement with Text-Consistent FiLM: Recover full-resolution masks using shallow features (stride 1 and 2) and apply lightweight FiLM conditioning from the global text vector at 112×112 and 224×224. This ensures pixel-level boundary sharpening while maintaining text-driven semantic consistency, reducing leakage and closing gaps in thin structures.

6. Learnable Gating for Frequency Components: Use a learnable gate (G(s)) to softly weight informative DCT coefficients when computing spectral energy, preventing noise from irrelevant high-frequency components. This improves robustness to image artifacts and enhances generalization across datasets.

7. Adaptive Text Length Handling: Implement a default text length of L=24 tokens (padded/truncated) to balance information richness and noise. The AI can truncate overly long clinical notes to retain lesion descriptors while ignoring redundant tokens, improving performance stability.

8. Combined Dice and Cross-Entropy Loss with Cosine Annealing: Optimize with a hybrid loss (Dice + CE) and AdamW with cosine annealing (lr=5×10−5, ηmin=10−6) over 160 epochs. This stabilizes training, accelerates convergence, and yields higher overlap (Dice) and boundary accuracy (HD95) compared to single-loss or fixed-lr schedules.

---

What the Improved AI System Can Do:

• Accurate Pulmonary Infection Segmentation: Achieve 91.46% Dice / 84.26% mIoU on chest X-rays (QaTa-COV19) and 81.95% Dice / 69.42% mIoU on CT scans (MosMedData+), outperforming prior baselines by +1.96 Dice and +2.67 mIoU on average.

• Text-Guided Focus: Interpret clinical text (e.g., bilateral ground-glass opacities with peripheral distribution) to selectively highlight infected regions, suppress healthy tissue, and adapt to varying lesion shapes and sizes.

• Boundary Precision: Reduce Hausdorff distance (HD95) by leveraging frequency-domain recalibration and high-resolution refinement, producing masks with sharp edges and minimal leakage into adjacent anatomy.

• Robustness to Input Variability: Handle truncated or verbose clinical notes gracefully (optimal at L=24), and maintain performance across different imaging modalities (X-ray vs. CT) without retraining.

• Efficient Deployment: Operate with 47.77M parameters and 20.31 GFLOPs, making it suitable for real-time clinical decision support on standard GPU hardware (e.g., NVIDIA RTX 3090Ti).

• Explainable Multi-Stage Refinement: Provide progressive intermediate outputs (7×7 → 56×56 → 224×224) that allow clinicians to visualize how the model moves from coarse localization to fine boundary delineation, aiding trust and interpretability.