Learn to Rank: Visual Attribution by Learning Importance Ranking
cs.CV, cs.LG
Submitted: 2026-04-07
Updated: 2026-09-05
Comments: ECCV 2026, code is available at https://github.com/dschinagl/AHA
Code: https://github.com/dschinagl/AHA
License: http://creativecommons.org/licenses/by/4.0/
The gist: Interpreting the decisions of complex computer vision models is crucial to establish trust and accountability, especially in safety-critical domains.
Terminology
Abstract
Interpreting the decisions of complex computer vision models is crucial to establish trust and accountability, especially in safety-critical domains. An established approach to interpretability is generating visual attribution maps that highlight regions of the input most relevant to the model's prediction. However, existing methods face a three-way trade-off. Propagation-based approaches are efficient, but they can be biased and architecture-specific. Meanwhile, perturbation-based methods are causally grounded, yet they are expensive and for vision transformers often yield coarse, patch-level explanations. Learning-based explainers are fast but usually optimize surrogate objectives or distill from heuristic teachers. We propose a learning scheme that instead optimizes deletion and insertion metrics directly. Since these metrics depend on non-differentiable sorting and ranking, we frame them as permutation learning and replace the hard sorting with a differentiable relaxation using Gumbel-Sinkhorn. This enables end-to-end training through attribution-guided perturbations of the target model. During inference, our method produces dense, pixel-level attributions in a single forward pass with optional, few-step gradient refinement. Our experiments demonstrate consistent quantitative improvements and sharper, boundary-aligned explanations, particularly for transformer-based vision models. Code and pretrained models are available at https://github.com/dschinagl/AHA.
Sources
- Use HiResCAM instead of Grad-CAM for faithful explanations of convolutional neural networks
- DINOv3
- Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models