SegPAR: Class-Centric Decision-Based Sparse Attack for Semantic Segmentation

SegPAR: Class-Centric Decision-Based Sparse Attack for Semantic Segmentation proposes a novel decision-based black-box sparse attack framework for semantic segmentation. The paper states: Despite the practical relevance of sparse decision-based blackbox threats, they have received limited attention in semantic segmentation. To address this gap, the authors adapt existing decision-based black-box sparse attacks from classification (PointWise, SparseEvo, RFPAR) as baselines, establishing a benchmark for this setting. They demonstrate that RFPAR suffers from severe query inefficiency due to its image-centric pixel accumulation, which rapidly exhausts query budgets across the vast image space.

The proposed method, SegPAR, shifts to a class-centric exploration paradigm. It constructs per-class binary masks from the initial prediction, extracts class-specific crops as agent states, and uses a mask-based sampling strategy to constrain perturbations to actual class regions. The paper introduces a discrepancy reward (Rd) to replace the standard decision reward (Rs), which provides misleading feedback: the agent receives positive reinforcement not only for desirable transitions in Snew but also for undesirable ones in Srevert and Sfluctuate. The discrepancy reward directly tracks marginal gains of cumulatively successful pixels, assigning r=0 for maintained misclassification, r=-1 for reversion, r=+1 for new misclassification, and r=0 for fluctuation.

Experiments evaluate SegPAR against PointWise, SparseEvo, and RFPAR on ADE20K, Pascal VOC2012, and Cityscapes using DeepLabV3, PSPNet, SegFormer, and SETR models under a 1,000-query budget. Results show SegPAR with Rd consistently outperforms existing sparse attack techniques across all evaluated datasets, achieving the lowest R.MIoU under extreme sparsity at comparable or lower queries. For example, it drops PSPNet's MIoU on Cityscapes to 0.057 with 3.53% sparsity, and achieves 2.24% sparsity on VOC2012 for DeepLabV3 with 680.42 average queries. Against adversarially trained models (DDCAT, SAT), SegPAR consistently attains the lowest R.MIoU with lower sparsity than prior methods. Per-class analysis shows SegPAR degrades safety-critical classes (traffic signs, traffic lights, persons, riders) much earlier than with RFPAR, with SegPAR driving their per-class MIoU down sharply within ∼200 queries. The discrepancy reward applied to Pixle and Sparse-RS also improves MIoU reduction on ADE20K and VOC2012, though on high-resolution Cityscapes it can stagnate alone; combining it with SegPAR's class-wise inputs alleviates this. Compared to white-box sparse attacks (PGD0, sPGD), SegPAR remains strongly competitive across both segmentation models—often outperforming PGD0 and approaching sPGD under strict sparsity constraints. The paper concludes that SegPAR achieves substantially larger MIoU degradation with fewer perturbed pixels than existing black-box baselines and highlights the persistent vulnerability of dense prediction models under realistic decision-only sparse threats.

Improvements to AI Systems:

1. Class-Centric Perturbation Constraint for Robustness Evaluation

• Improvement: Integrate SegPAR’s class-specific binary mask generation and crop-based state extraction into adversarial training pipelines. This allows models to be trained against attacks that target only semantically meaningful regions (e.g., traffic signs, pedestrians) rather than uniform image noise.

• Capability: The improved system can identify and harden the most safety-critical object classes first, reducing the risk of catastrophic misclassification in autonomous driving or surveillance under sparse, decision-only attacks.

2. Discrepancy Reward for Reinforcement-Learning-Based Attack/Defense

• Improvement: Replace standard decision rewards (which reward both desirable and undesirable state changes) with the discrepancy reward (Rd) in any RL-based adversarial agent. This prevents the agent from being misled by fluctuations or reversions, focusing optimization on cumulative, irreversible pixel changes.

• Capability: The improved system can generate more query-efficient attacks (or defenses) that require fewer iterations to achieve the same misclassification, especially in high-dimensional outputs like segmentation maps, where noisy feedback is common.

3. Query-Budget-Aware Adaptive Sparse Attack Scheduler

• Improvement: Adopt SegPAR’s per-class query allocation strategy, which prioritizes classes with higher initial confidence or larger spatial extent, into any black-box attack framework. This prevents premature exhaustion of the 1,000-query budget on background or large static regions.

• Capability: The improved system can operate effectively under strict query limits (e.g., <500 queries) on high-resolution inputs, maintaining high attack success while perturbing <3% of pixels—useful for real-time adversarial testing in cloud-based vision APIs.

4. Cross-Model Transferable Sparse Attack Generator

• Improvement: Use SegPAR’s class-centric masks and discrepancy reward to train a meta-attack generator that transfers across segmentation architectures (DeepLabV3, PSPNet, SegFormer, SETR) without retraining. The generator learns class-agnostic perturbation patterns that generalize across model families.

• Capability: The improved system can pre-emptively test a suite of deployed segmentation models against a single, fast sparse attack, enabling rapid vulnerability screening before deployment—reducing the need for per-model adversarial evaluation.

5. Per-Class Robustness Monitoring and Early Warning

• Improvement: Implement SegPAR’s per-class MIoU degradation tracking (e.g., traffic signs drop sharply within 200 queries) as a real-time monitoring metric in production segmentation systems. This flags when a model becomes vulnerable to sparse attacks on specific classes before full-system failure.

• Capability: The improved system can trigger automatic retraining or fallback mechanisms for safety-critical classes (e.g., pedestrians, traffic lights) when their robustness drops below a threshold, enhancing operational safety in autonomous vehicles or medical imaging.

6. Hybrid Sparse Attack Ensemble for High-Resolution Inputs

• Improvement: Combine SegPAR’s class-wise input cropping with the discrepancy reward applied to existing sparse attacks (Pixle, Sparse-RS) to overcome stagnation on high-resolution images (e.g., Cityscapes). This hybrid uses class crops to focus the attack and Rd to avoid local minima.

• Capability: The improved system can effectively attack ultra-high-resolution segmentation models (e.g., 2048x1024) with <4% sparsity, where prior methods fail due to query inefficiency—enabling robust evaluation of state-of-the-art dense predictors in real-world settings.

7. White-Box Sparse Attack Enhancement via Class-Centric Priors

• Improvement: Incorporate SegPAR’s class-centric perturbation constraints into white-box sparse attacks (PGD0, sPGD) to reduce search space and improve convergence under strict sparsity (e.g., <1% pixels).

• Capability: The improved system can generate near-optimal sparse perturbations faster than existing white-box methods, useful for generating certified robustness benchmarks or adversarial examples for model interpretability.