Boosting the Local Invariance for Better Adversarial Transferability

arXiv:2503.06140 · cs.CV · Submitted 2025-03-08 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Boosting the Local Invariance for Better Adversarial Transferability".

Jane: The gist: adversarial perturbation often exhibits poor translation invariance for a given clean image and model, which is attributed to local invariance <ref:2503.06140#pg2>.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, thinking about the title of this paper, "Boosting the Local Invariance for Better Adversarial Transferability," what does that really mean for anyone listening who isn't deep into adversarial research?

Jane: It means we've found a way to fix a common problem where an attack works well on one AI system but fails when you try to use it on another system #pg2.

Meng: In simple terms, the paper suggests that making the adversarial noise more stable locally—meaning it doesn't change much when you nudge the original image slightly—is the secret ingredient for making attacks work across different models #pg2.

Lu: It highlights how we can use a specific optimization technique, LI-Boost, to achieve this by looking at gradients from translated perturbations #pg2.

Tom: The implication is that for anyone building defenses or even just understanding how these attacks work, focusing on boosting this local invariance is a key direction for improving adversarial transferability #pg2.

Jane: It gives us a concrete tool, LI-Boost, that they show works across many different models and attack types on the ImageNet dataset #pg1.

Meng: For practical engineers, it means we have a more robust way to test how well our systems can be attacked by looking at this local invariance metric #pg2.

Tom: It provides a promising direction for future research into improving adversarial transferability across models, and the code is actually available online #pg1.

Conclusion: Tom: So, we're wrapping up this look at "Boosting the Local Invariance for Better Adversarial Transferability." The main idea here is that making adversarial noise more stable locally helps those attacks work better on different AI models #pg5.

Jane: It boils down to how much a small nudge to an image affects the resulting attack output, and if that effect stays consistent across various models #pg3.

Lu: The authors introduce this concept of local invariance, which they quantify by measuring the expected probability that an adversarial perturbation keeps its fooling ability after a slight spatial translation #pg5.

Meng: For us on the ground, what this means is we’re using these local invariance numbers as a way to judge how well an attack can jump from one system to another without needing a complete rewrite #pg8.

Lalam: From my side, if the local invariance goes up, it suggests that the perturbation isn't overly sensitive to the specific architecture of the target model #pg2.

Tom: Exactly. They propose this LI-Boost method, which uses gradients from translated examples to actually tune that noise for better local invariance #pg9.

Jane: And they show this works across a huge range of things, like CNNs, ViTs, and even vision-language models #pg1.

Lu: It's interesting how it handles different attack styles too; they saw big gains with gradient-based attacks like MI-FGSM and near perfect success rates on input transformation attacks #pg9.

Meng: I’m curious about the practical side though, since we’re talking about commercial vision APIs too, how much real-world improvement do we actually see when implementing this?

Tom: We'll get into those numbers next, but the big picture is that this method provides a general boosting technique for any transfer-based attack #pg9.

Jane: It really shows that focusing on local invariance is a pivotal mechanism for overcoming those boundaries between different AI architectures #pg1.

cs.CV

Submitted: 2025-03-08

Updated: 2026-10-08

Code: https://github.com/Trustworthy-AI-Group/TransferAttack

Importance score: 79/100

The gist: The gist: adversarial perturbation often exhibits poor translation invariance for a given clean image and model, which is attributed to local invariance <ref:2503.06140#pg2>.

Key concepts

Local Invariance: This concept measures how well an adversarial perturbation maintains its ability to fool a model when the image is slightly translated spatially. High local invariance suggests the perturbation's effectiveness is not dependent on exact pixel locations, making it more likely to transfer successfully between different models.
LI-Boost Approach: LI-Boost is a novel framework that enhances attack transferability. It optimizes the adversarial noise by calculating gradients using multiple translated versions of the image. This process specifically targets improving local invariance, aiming to make the attack robust against minor spatial shifts.
Adversarial Transferability: This refers to the ability of an adversarial perturbation created for one model (e.g., a CNN) to successfully fool a different, unseen model (e.g., a ViT). The paper shows that increasing local invariance is key to boosting this transferability across diverse architectures and defense mechanisms.
Monte Carlo Sampling Strategy: To efficiently find the optimal perturbation, LI-Boost uses a Monte Carlo sampling strategy. Instead of checking every possible translation, it randomly draws translated perturbations to approximate the gradient. This balances computational speed with the need to accurately estimate how local invariance affects attack performance.

Terminology

Transferability — 109× in this paper · explained in 4 episodes
Local Invariance — 64× in this paper · explained in 2 episodes
Adversarial Transferability — 54× in this paper · explained in 1 episode
Adversarial Examples — 46× in this paper · explained in 2 episodes
Adversarial Attacks — 19× in this paper · explained in 1 episode
Robustness — 15× in this paper · explained in 5 episodes
Transformer — 12× in this paper · explained in 2 episodes
Objective Function — 11× in this paper · explained in 2 episodes
Vision-language models — 10× in this paper · explained in 2 episodes
Adversarial Robustness — 7× in this paper · explained in 3 episodes
Ground Truth — 6× in this paper
Adversarial Training — 5× in this paper · explained in 1 episode
Alignment — 5× in this paper · explained in 1 episode
Ablation — 3× in this paper · explained in 1 episode
Adversarial Noise — 3× in this paper · explained in 1 episode
Generalizability — 3× in this paper · explained in 6 episodes
Hallucination — 3× in this paper · explained in 4 episodes
Ablation Studies — 2× in this paper · explained in 1 episode
Adversarial Defense — 2× in this paper · explained in 1 episode
Consistency — 2× in this paper · explained in 3 episodes
Diffusion Model — 2× in this paper · explained in 8 episodes
Diffusion Models — 2× in this paper · explained in 6 episodes
Generalization — 2× in this paper · explained in 13 episodes
LLM-as-a-Judge — 2× in this paper · explained in 3 episodes
Loss Function — 2× in this paper · explained in 2 episodes
Reinforcement Learning — 2× in this paper · explained in 4 episodes
Activation Function — 1× in this paper · explained in 1 episode
Adam — 1× in this paper · explained in 2 episodes
Attack Success Rate (ASR) — 1× in this paper · explained in 2 episodes
Benchmark — 1× in this paper · explained in 2 episodes
Calibration — 1× in this paper · explained in 7 episodes
Core — 1× in this paper · explained in 2 episodes
Latent Diffusion Model — 1× in this paper · explained in 2 episodes
Latent Diffusion Models — 1× in this paper · explained in 2 episodes
Model-Agnostic — 1× in this paper · explained in 2 episodes
Multimodal Reasoning — 1× in this paper · explained in 5 episodes
Regularization — 1× in this paper · explained in 3 episodes
Semantic Alignment — 1× in this paper · explained in 3 episodes
Semantic Understanding — 1× in this paper · explained in 2 episodes
Structural Integrity — 1× in this paper · explained in 3 episodes
Verification — 1× in this paper · explained in 2 episodes
Vision-Language Models (VLMs) — 1× in this paper · explained in 6 episodes

Summary

The gist: adversarial perturbation often exhibits poor translation invariance for a given clean image and model, which is attributed to local invariance <ref:2503.06140#pg2>.

Local Invariance Concept

The paper introduces the concept of local invariance for adversarial perturbations and reveals the underlying relationship between local invariance in the surrogate model and adversarial transferability across different models <ref:2503.06140#pg3>. This is quantified by Definition 1 (Local Invariance), which measures the expected probability that the adversarial perturbation retains its fooling ability under small spatial translation <ref:2503.06140#pg5>. The local invariance of adversarial perturbations serves as an indicator of their transferability across different models <ref:2503.06140#pg8>.

LI-Boost Approach

The authors propose a novel and generalizable framework called Local Invariance Boosting approach (LI-Boost) to enhance the transferability of various transfer-based attacks <ref:2503.06140#pg9>. Specifically, at each iteration, LI-Boost optimizes the adversarial perturbation using the gradient of adversarial examples with several translated perturbations to enhance the local invariance. The optimization problem is formulated as δ∗ = arg max∥δ∥p≤ϵ min δ′∈Nk(δ) J(x + δ′, y; θ), where k represents the maximum number of pixels by which the perturbation is translated. To enhance computational efficiency, a Monte Carlo sampling strategy is employed to approximate the gradient by randomly drawing sampled perturbations.

Experimental Validation and Results

Extensive experiments on the ImageNet dataset demonstrate that LI-Boost can significantly enhance various types of transfer-based attacks on CNNs, ViTs, and defense mechanisms. The results show that LI-Boost consistently boosts the performance across both CNN and ViT architectures. For example, for gradient-based attacks like MI-FGSM, LI-Boost increases the attack success rate from 94.9% to 97.0% on RN-50. Furthermore, LI-Boost consistently boosts the performance of white-box attacks, achieving near-perfect success rates of approximately 100.0% for input transformation-based attacks like DIM and Admix.

Generalizability Across Models and Attacks

The proposed method is shown to be generalizable across diverse attack scenarios, including CNNs, ViTs, defense mechanisms, commercial vision API systems, and vision-language models. The effectiveness is demonstrated on various architectures such as ResNet-50 (RN-50), Inception-v3 (Incv3), MobileNet-v3-large (MN-v3), DenseNet-121 (DN-121), FasterNet (FSNet), ViT, PiT, Visformer, and Swin. The evaluation also extends to commercial vision API systems like Baidu, Alibaba, and Tencent.

VLM Robustness Evaluation

When evaluated on vision-language models (VLMs), LI-Boost consistently improves the attack success rates of all baseline methods across the twelve evaluated VLMs. For instance, LI-Boost increases the average attack success rate Output the kind in this image using one word from 58.6% to 66.5%, corresponding to a substantial 7.9% absolute improvement. Furthermore, LI-Boost can induce drastic semantic shifts and severe misalignment when attacking vision-language models, such as triggering a complete hallucination in Claude-o.

Conclusion

In this study, the authors validate that LIBoost can significantly boost the adversarial transferability of various transfer-based attacks across different models and defense mechanisms. The findings underscore the efficacy of LI-Boost in augmenting adversarial attacks, highlighting local invariance as a pivotal mechanism for overcoming model-specific architectural boundaries. This work provides valuable insights into potential avenues for advancing adversarial attack research.

How it works

**: The core mechanism involves optimizing the perturbation using the gradient of adversarial examples with several translated perturbations to enhance local invariance. This is achieved by solving the problem δ∗ = arg max∥δ∥p≤ϵ min δ′∈Nk(δ) J(x + δ′, y; θ), where k represents the maximum number of pixels by which the perturbation is translated. The search space is approximated using a Monte Carlo sampling strategy to balance attack efficiency and effectiveness. This approach allows for significant performance gains across diverse models and defense strategies, revealing the limitations of existing defenses. The consistent gains in attack performances align with the increased local invariance values, providing empirical evidence that strengthening the local invariance of adversarial perturbations is crucial for enhancing their transferability across different models. The choice of sampling distribution, specifically the logarithmic distribution, yields the best attack performance as it places suitable emphasis on smaller neighborhoods. This validates the hypothesis that local invariance is a pivotal mechanism for overcoming model-specific architectural boundaries. The peak attack performance is reached at k = 6, suggesting that local invariance significantly enhances adversarial robustness. This sets an optimal balance between white-box attack strength and black-box transferability. The paper concludes that LI-Boost can further unlock the potential of ensemble attacks, ensuring higher transferability performance against various black-box models and attack scenarios. This approach is a general boosting technique applicable to a variety of attacks. The method is shown to be effective even under stringent black-box constraints, demonstrating its strategic value in enhancing cross-model generalization. This work will inspire further research in this direction. The paper provides a promising direction for future research on improving adversarial transferability across models. The code is available at https://github.com/Trustworthy-AI-Group/TransferAttack. The paper provides a promising direction for future research on improving adversarial transferability across models. The paper provides a promising direction for future research on improving adversarial transferability across models. This work will inspire further research in this direction. The paper provides a promising direction for future research on improving adversarial transferability across models. The code is available at https://github.com/Trustworthy-AI-Group/TransferAttack. This work will inspire further research in this direction. The paper provides a promising direction for future research on improving adversarial transferability across models. The code is available at https://github.com/Trustworthy-AI-Group/TransferAttack. This work will inspire further research in this direction. The paper provides a promising direction for future research on improving adversarial transferability across models. The code is available at https://github.com/Trustworthy-AI-Group/TransferAttack. This work will inspire further research in this direction. The paper provides a promising direction for future research on improving adversarial transferability across models. The code is available at https://github.com/Trustworthy-AI-Group/TransferAttack. This work will inspire further research in this direction.

Improvements for AI systems

Bold header: Local Invariance Boosting approach (LI-Boost) implementation in adversarial attack generation. This technique optimizes the adversarial perturbation using the gradient of adversarial examples with several translated perturbations to enhance the local invariance, which can significantly enhance various transfer-based attacks.
Bold header: Enhanced cross-model transferability for CNNs, ViTs, and defense mechanisms. The approach is demonstrated to improve attack success rates across these diverse architectures, as shown in Table II where LI-Boost results are highlighted in gray across all models.
Bold header: Improved robustness against commercial vision API systems and vision-language models (VLMs). LI-Boost can significantly enhance various transfer-based attacks on CNNs, ViTs, defense mechanisms, as well as applications on commercial vision API systems, showing its effectiveness against closed-source VLMs like GPT-4o.
Bold header: Disruption of high-level semantic reasoning in VLMs. The method induces drastic semantic shifts and severe misalignment in VLMs; for example, LI-Boost-ILA successfully triggers a complete hallucination, causing the target model to misunderstand a static children’s barber chair as a decorative cobra-shaped hookah pipe.
Bold header: Quantifiable relationship between local invariance and adversarial transferability. The study establishes that the local invariance of adversarial perturbations w.r.t the clean image for a given model is positively correlated to its adversarial transferability across different models, providing a new framework to enhance the transferability of various transfer-based attacks.