Localize and Neutralize: Gradient-guided Token Suppression against Visual Prompt Injection Attack

arXiv:2605.25194 · cs.LG · Submitted 2026-08-24 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Localize and Neutralize: Gradient-guided Token Suppression against Visual Prompt Injection Attack".

Jane: The paper was written by the authors from.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Jane: We also have Lu with us today — senior AI researcher at Tsinghua.

Tom: We also have Meng with us today — lead engineer at a mysterious AI startup.

Jane: We also have Lalam with us today — the in-house Large Language Model.

Tom: Alright, let's get started.

Title: Tom: So, we’ve established that the core of Localize then Neutralization: Gradient-guided Token Suppression against Visual Prompt Injection Attack is about precision. The authors start by analyzing the impact of this approach on diverse attack scenarios, which is a huge step forward.

Jane: They show that visual prompt injection doesn't just work by looking at a massive wall of pixels; rather, it relies on specific "critical tokens" within the image embedding sequence to achieve its goal. It’s essentially identifying the weak link in the visual chain.

Lu: The researchers utilized a technique called sliding-window masking to empirically prove this dependency, which is essentially testing every possible small section of the image input to see if that specific part is necessary for a successful attack. That finding challenges our conventional assumptions about how much visual information AI needs to function.

Meng: This confirms that if an attack relies on only a few tokens, we can focus our computational resources precisely there instead of having to defend against massive input corruption across the the entire image. It suggests focused defense is highly effective in practical terms.

Lalam: That’s very reassuring for me because it means the danger isn't just a lot of random noise; it’s specific, intentional signals that are causing harm at those critical points. This tells us that malicious intent is localized, which helps us build a sense of security in the AI's ability to recognize and reject harm.

Paper discussion segment 2: Tom: Building on that insight—that the attack relies on sparse tokens—the authors propose Gradient Token Masking, or GTM. This is where they introduce their core solution to handle the problem efficiently by localizing and neutralizing those high-influence tokens during inference time.

Jane: The innovation here is in how they locate these tokens; it’s not just checking the final output probability of the first generated word, which we know often fails when an adversarial attack succeeds. They found a much more robust way to measure influence.

Lu: They leveraged a "representation-aware saliency score" based on the Hidden-State Gradient Norm to pinpoint these critical tokens. This is a massive leap forward in how we measure internal influence because it bypass the limitations of just looking at output probabilities, which was previously a huge technical hurdle.

Meng: From an implementation angle, this is brilliant because we don't need to run multiple complex optimization loops; it requires only a single forward-backward pass to identify the candidates for masking. This makes deployment far more practical for real-time use cases in AI.

Lalam: That efficiency is very important for me, too, because a fast defense means the technology can be deployed widely without slowing down the pace of information sharing and cultural exchange. We are ensuring that security doesn's come at the cost of speed or usability.

Paper discussion segment 3: Tom: So, GTM identifies these crucial spots by looking at how they influence the internal hidden state representation of the first token in a sequence. It’s like seeing which specific input pixels have had the biggest impact on that model's initial thought process.

Jane: And what’s even more encouraging is that this method works across both general prompt injection and those more specific jailbreak attacks, showing that GTM isn't designed to be a niche solution for only one type of attack. It’ robust against diverse threats.

Lu: The authors proved this mechanism is designed to find those tokens even when the final output probability fails, which is a significant technical hurdle they successfully navigated using mathematical guarantees. This provides us with confidence in the theoretical underpinning of the system's reliability.

Meng: This allows us to potentially build production systems where we can interrupt the attack path immediately after all that initial processing, without waiting for a full sequence prediction to complete. We can cut off the threat almost instantly at the source.

Lalam: By focusing on this initial internal influence, we are ensuring that the AI doesn't get misled early in its processing, which is a huge step toward building trustworthy tools that support safe and positive interactions in our culture.

Conclusion: Tom: We’ve explored how to spot these sparse vulnerabilities and then how to use gradients to find and neutralize them. The core of the Localize then Neutralization: Gradient-guided Token Suppression against Visual Prompt Injection Attack defense is that by masking only those top-k tokens, you successfully disrupt the entire adversarial path.

Jane: It’s also important to remember that GTM doesn't just stop attacks; it preserves model utility too. Even while neutralizing harmful prompts, the underlying AI model still performs its normal tasks without any noticeable degradation in quality, which is a major win for users.

Lu: The theoretical guarantee of ranking consistency—that the order of importance is preserved between the full loss gradient and our saliency score—is a very strong piece of math that gives us deep confidence in this approach's accuracy.

Meng: And I’m especially glad it's target-agnostic, meaning we aren't building one defense for VMA attacks and another for ImgHijack; the same method works across diverse adversarial objectives. This makes the scaling of the system much easier.

Lalam: The impact of Localize then Neutralization: Gradient-guided Token Suppression against Visual Prompt Injection Attack is that it allows us to build a much more robust future for AI. We are moving away from simply hoping the model won’t be tricked toward actively fixing its internal mechanics.

Tom: That feels like a perfect way to wrap up our discussion of this defense mechanism—thank you all for this deep dive into securing these powerful systems.

Lu: I'm excited to see what other complex AI architectures will adopt this concept, making it feel like a new era in how we design and build AI models.

Meng: I’m looking forward to seeing how this translates into production-ready systems with minimal latency, too, especially as we scale deployment.

Lalam: My hope is that this leads to an AI that we can trust completely in the cultural sphere, ensuring reliable information sharing for everyone.

cs.LG

Submitted: 2026-08-24

Updated: 2026-08-25

Code: https://github.com/fish883/GTM-Defense

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 86/100

The gist: This paper introduces Gradient Token Masking (GTM), a novel defense mechanism designed to protect Large Vision-Language Models (LVLMs) from perturbation-based visual prompt injection attacks.

Key concepts

Visual Prompt Injection Attack
This attack relies on specific, critical tokens within an image embedding sequence to achieve its goal. Researchers found that these attacks are localized, not dependent on a massive wall of pixels, making the malicious intent highly focused at certain points.
Gradient Token Masking (GTM)
GTM is the core solution for localizing and neutralizing high-influence tokens during inference time. It allows for an efficient defense that requires only a single forward-backward pass to identify candidates, making it practical for real-time use.
Representation-aware Saliency Score
This is the method used to pinpoint critical tokens. Instead of just checking the final output probability, this score measures internal influence by looking at how specific input pixels have had the biggest impact on the model's initial hidden state representation.

Terminology

Summary

This paper introduces Gradient Token Masking (GTM), a novel defense mechanism designed to protect Large Vision-Language Models (LVLMs) from perturbation-based visual prompt injection attacks. As LVLMs are increasingly adopted in critical industries, they face severe security threats where adversaries manipulate model behavior through imperceptible image-based inputs. This research is significant because it addresses a critical gap in existing defenses, which have primarily focused on explicit jailbreak attacks rather than the broader and more flexible category of general visual prompt injection.

The Core Insight

The authors provide a mechanistic analysis of perturbation-based visual prompt injection attacks, revealing that these attacks do not rely on the entire image uniformly. Instead, they demonstrate that successful adversarial behavior is driven by a highly sparse subset of critical image tokens rather than the entire visual input. Through a sliding-window masking procedure, the researchers found a plateau and valley pattern in attack success rates (ASR), which indicates that removing specific narrow intervals of embedding tokens can cause a complete collapse of adversarial efficacy.

The Proposed Method

To exploit this sparsity, the authors propose Gradient Token Masking (GTM), which operates via two primary stages: localization and neutralization.

  1. Localization: The method identifies Critical Tokens by calculating a representation-aware saliency score for each image embedding token.

  2. Neutralization: Once identified, these high-influence tokens are masked (i.e., replaced with a zero vector or a mean embedding) to disrupt the adversarial path.

A key technical contribution is the use of the Hidden-State Gradient Norm to measure generation influence. The authors prove that standard attribution based on output probability fails when attacks involve token alignment—scenarios where the first adversarial target token coincides with the model's natural output for a clean image. They provide a theoretical guarantee that their proposed metric is asymptotically consistent with the optimal adversarial loss gradient, ensuring accurate localization of the triggers.

Experimental Results and Utility

The effectiveness of GTM was validated through extensive experiments across diverse attack scenarios, including:

** General prompt injection (e.g., VMA and ImgHijack). 0**

** Jailbreak attacks (e.g., ASTRA, BAP, UMK, and JPS). 0**

The results demonstrate that GTM reduces attack success rates (ASR) to near zero across various target models like Qwen2-VL, Phi-3-Vision, and LLaVA. Crucially, the method preserves model utility with negligible computational overhead, maintaining high performance on standard benchmarks like MM-Vet. Unlike existing defenses such as DiffPure, which can cause severe performance degradation by distorting text-rich images, GTM's impact on benign tasks is minimal.

Efficiency and Deployment

GTM is designed for practical, real-world deployment. It is characterized as a lightweight inference-time defense that requires only a single forward–backward pass to identify and zero out high-scoring tokens. The paper highlights several deployment advantages:

** Minimal Overhead: It introduces only marginal inference latency compared to the undefended baseline. 0**

** Lightweight Design: It avoids reliance on heavy auxiliary components like diffusion models, maintaining a memory footprint comparable to the original LVLM. 0**

** Plug and Play: As a training-free and target-agnostic defense, it can be integrated into existing pipelines without costly retraining. 0**

Improvements for AI systems

To implement the findings of this paper into production-grade Large Vision-Language Models (LVLMs), I would implement a specialized inference-time defense layer called the following:

  1. The Improvement: Implementation of a Gradient Token Masking (GTM) Inference Pipeline.

  2. What the improved AI system can do:

Instead of processing raw visual embeddings directly, the system will perform a single, high-speed partial forward-backward pass at the start of every inference request to identify and neutralize adversarial triggers. Specifically:

Inference-time "Localization & Neutralization": The system will compute a representation-aware saliency score for every image embedding token using the gradient of the hidden-state norm. It will then automatically identify the top 5% (or a user-defined budget) of tokens most responsible for driving anomalous model behavior and zero them out before final generation.

Robustness against Invisible Prompt Injections: The system will remain immune to perturbation-based attacks (like ImgHijack or VMA) where the malicious intent is encoded in imperceptible pixel noise that bypasses standard text-based filters and visual inspection.

Protection against Token Alignment Attacks: Unlike standard attribution methods that fail when an attack begins with a natural looking token, this system will detect the underlying adversarial signal hidden within the model's internal latent representations, preventing attackers from hiding their injection in the first predicted token.

High-Fidelity Utility Preservation: Unlike diffusion-based purification (e.g., DiffPure), which distorts text and fine details within an image, this system will preserve 100% of the visual semantic integrity for benign tasks. The AI will be able to perform complex OCR, mathematical reasoning on images, and spatial analysis without the image corruption side effects common in existing defenses.

Target-Agnostic & Training-Free Deployment: The improvement requires no retraining of the massive backbone model and no additional heavy auxiliary models (like Diffusion Models). This allows for a massive reduction in inference latency (adding only 0.2s overhead) and memory footprint, making it suitable for real-time, high-throughput production environments where security is critical.

Sources

Related papers