Entropy-Centric Explainable AI for Remote Sensing Image Segmentation

This paper proposes an entropy-centric explainable AI (XAI) method for semantic segmentation in remote sensing imagery, addressing the lack of transparency in deep neural network decision-making. The authors state: "this paper proposes an entropy-centric XAI method for semantic segmentation. Moreover, a new XAI evaluation methodology is proposed to efficiently measure the relevance of the highlighted regions by the proposed XAI method."

The proposed method builds on the conventional Sobol XAI method, which uses variance-based global sensitivity analysis with quasi-Monte Carlo (QMC) sampling and the Jansen estimator to compute total Sobol indices. The authors adapt this to image segmentation by shifting from explaining scalar outputs to handling spatially structured outputs. The method works by: (1) initializing base sampling matrices A and B using QMC, (2) generating perturbation masks Ci, (3) creating perturbed images by applying upsampled masks to the input image, (4) performing model inference to obtain spatial score maps, (5) masking the target object using the object's mask M, (6) computing softmax probabilities, and (7) calculating importance scores based on binary entropy changes. The binary entropy H is defined as: H = -pk log(pk) - (1-pk) log(1-pk), where pk is the predicted target class probability at each pixel. The importance score is calculated as: Si = (1/N) Σ [H(PCi,j) - H(PAj)], where H(Cj) and H(Aj) denote entropy values for perturbed and base samples. The authors note that "the proposed Entropy-Centric XAI method focuses on the uncertainty which offers a different explainability approach than the Sobol sensitivity method by revealing areas degrading the certanity of the model after perturbation."

The paper also proposes a new XAI evaluation methodology called High-Salience Influence Test (H-SIT), which complements the existing Low-Salience Irrelevance Test (L-SIT). While L-SIT removes pixels with explanation values below a threshold to test if low-importance regions are truly irrelevant, H-SIT removes high-salience regions outside the target object to assess whether they are truly influential. The perturbed image for H-SIT is defined as: I'H-SIT = T ∪ ϕ, where ϕ is the set of highlighted pixels below a specific threshold. The authors explain: "Through the proposed H-SIT XAI evaluation methodology, a large drop in the considered metric is expected since the regions most affecting the model decision are removed, enabling the identification of falsely highlighted regions that L-SIT methodology alone cannot capture."

Performance evaluation was conducted using the WHU dataset for building footprint segmentation with a rooftop U-Net architecture. The proposed Entropy-Centric method was compared against Grad-CAM, Score-CAM, and Seg-Sobol methods. Qualitative results show that Score-CAM and Entropy-Centric produce good explanation heatmaps by highlighting building areas and their surroundings, while Grad-CAM considers only small portions of the building pixels as important. Score-CAM exhibited failure cases such as highlighting wide regions far from the buildings and skip buildings and highlight different objects, while Entropy-Centric did not exhibit similar failure cases.

Quantitative results at a 0.1 threshold show that for L-SIT, the proposed Entropy-Centric method outperforms all benchmarks with only 2% prediction confidence drop, 0.7% IoU score drop, and 3.4% entropy score increase, compared to Seg-Sobol (12.2%, 12%, 18%), Grad-CAM (27.3%, 29.7%, 38.8%), and Score-CAM (0.6%, 4.1%, 8.9%). For H-SIT, Entropy-Centric achieves the largest drops: 37.4% prediction confidence drop, 44% IoU drop, and 48.5% entropy increase, demonstrating its ability to accurately isolate decision-critical regions. The authors note that Grad-Cam method barely has any drop in its metrics, which demonstrates its heatmap's deficiency in giving any importance to the regions outside the building areas.

The paper also provides a comparative analysis between Score-CAM and Entropy-Centric, noting that Score-CAM requires access to internal activation maps of the trained model, making it sensitive to architectural details and its implementation often relies on model-specific hooks and careful layer selection, while Entropy-Centric operates in a fully black-box manner and relies on systematic input perturbations, resulting in substantially improved stability and does not depend on gradients or internal representations.

The authors conclude that the proposed Entropy-Centric method is able to effectively isolate irrelevant regions as well as accurately identify decision-critical areas, establishing a robust foundation for XAI methods for semantic segmentation. Future directions include multi-dataset validation, hybrid XAI methods, and diverse architectures support.

Improvements to AI Systems:

1. Uncertainty-Aware Explainability for Segmentation Models: Integrate the entropy-centric perturbation framework into semantic segmentation systems to generate explanations based on predictive uncertainty rather than gradients or internal activations. This allows the AI to highlight not just where it is confident, but which input regions, when perturbed, increase its uncertainty—revealing fragile decision boundaries.

2. Black-Box Model Auditing for Spatial Outputs: Replace gradient-dependent explanation methods (e.g., Grad-CAM, Score-CAM) with the proposed fully black-box, Sobol-based entropy approach. This enables auditing of any segmentation model (including proprietary or non-differentiable systems) without architectural access, improving trust in third-party or legacy AI deployments.

3. Robust Saliency Validation via Dual Testing (L-SIT + H-SIT): Incorporate both Low-Salience Irrelevance Test and High-Salience Influence Test into model evaluation pipelines. This dual-check ensures that highlighted regions are not only necessary (removing them causes performance drops) but also sufficient (low-salience regions are truly irrelevant), reducing false positives in explainability and improving model debugging.

4. Entropy-Based Feature Importance for Active Learning: Use the entropy-centric importance scores to guide data selection in active learning loops. The AI can prioritize labeling pixels or images where perturbation-induced entropy changes are highest, accelerating learning on decision-critical, uncertain regions rather than random or confident samples.

5. Stable, Architecture-Agnostic Model Interpretation: Deploy the proposed method as a drop-in replacement for activation-map-based explainers in production systems. The improved AI can generate consistent explanations across different backbone architectures (U-Net, transformers, etc.) without retuning hooks or layer selections, enhancing cross-model comparability and reproducibility.

6. Failure-Case Detection for Segmentation Systems: Leverage the entropy-centric heatmaps to automatically flag regions where the model’s decisions are overly sensitive to small perturbations (high entropy change). This enables proactive identification of spurious correlations or over-reliance on non-target background features, leading to more robust and generalizable segmentation models.

What the Improved AI System Can Do:

• Provide transparent, uncertainty-driven explanations for semantic segmentation that are more faithful than gradient-based methods, especially in remote sensing and other high-stakes imagery domains.

• Operate on any segmentation model as a black box, requiring only input-output access, thus enabling auditing of commercial or closed-source systems.

• Automatically validate its own explanations using both removal and addition tests, ensuring that highlighted regions are truly decision-critical and non-highlighted regions are truly irrelevant.

• Prioritize data labeling and model fine-tuning on the most uncertain and decision-sensitive image regions, improving sample efficiency and model accuracy.

• Detect and mitigate hidden biases (e.g., focusing on roads instead of buildings) by revealing which external regions drive predictions, leading to more reliable AI in geospatial analysis, medical imaging, and autonomous driving.

Artificial intelligence (AI) has become a powerful approach to solving complex problems in critical domains. Many concerns arise regarding the decision-making process of its models, mainly due to deep neural networks outperforming their peers at the cost of ambiguity in feature extraction and prediction. Consequently, in critical domains such as remote sensing, where high-resolution imagery must be analyzed using black-box models, the lack of transparency limits trust in these models and, thus, their adoption. In light of this reality, explaining and understanding the complex decision-making process of AI models has become essential. Explainable AI (XAI) aims to bridge this gap by providing insights into how and why certain decisions are made. While significant progress has been achieved in explaining image classification tasks, image segmentation still offers considerable room for improvement. In this context, this paper proposes an entropy-centric XAI method for semantic segmentation. Moreover, a new XAI evaluation methodology is proposed to efficiently measure the relevance of the regions highlighted by the proposed XAI method. Experimental results demonstrate the superiority of the proposed XAI method compared with recently adapted XAI methods for semantic segmentation.