Once Poisoned, Arbitrarily Controlled: A Programmable Backdoor in VLMs

This paper introduces the first any-to-any backdoor attack on Vision-Language Models (VLMs), moving beyond the static one-to-one and N-to-N attack paradigm. The authors propose a programmable backdoor that allows an attacker to dynamically select previously unseen target-caption semantics at inference time and synthesize corresponding stealthy triggers on demand, without retraining the victim VLM.

The method has two core components. First, a heuristic poisoning strategy exposes the model to diverse trigger–caption pairs during training, encouraging it to learn a general trigger-as-instruction rule rather than memorizing a specific backdoor pattern. The poisoned dataset is constructed by overlaying a resized trigger image onto the top-left corner of a benign image and replacing the original caption with the trigger image's caption. Second, a feature-space trigger steganography method T S(·) maps any attacker-specified target caption z to a stealthy visual trigger, implemented as either a norm-controlled perturbation (with L∞ norm constraints of 8/255 or 16/255) or a non-semantic patch. This optimization uses a pre-trained vision feature extractor Φ to mimic the features of a vanilla trigger image t(z) while remaining imperceptible.

Experiments are conducted on LLaVA-1.6, Qwen3-VL, and MiniGPT-v2 models using Flickr8k and Flickr30k datasets, with trigger-target datasets derived from Tiny-ImageNet and CIFAR-100. Results show high any-to-any caption-control success rates: the vanilla trigger achieves 92.00% ASR on Flickr8k, the L∞ (ϵ=16/255) trigger achieves 86.00%, and the patch trigger achieves 87.67%. The attack preserves clean model utility, with poisoned models showing comparable performance to clean models on GQA and MME benchmarks. The generated triggers do not cause unintended activation on clean models, confirming stealthiness.

Compared to baselines, TrojVLM and MTAttack achieve near-perfect ASR on seen trigger-target pairs but 0% on held-out pairs, while IAG drops to 37.33-49.67% on unseen pairs. In contrast, the proposed method retains 92.00% and 86.00% ASR on unseen pairs with vanilla and L∞ triggers respectively. The attack also demonstrates robustness against classical backdoor defenses including Shrinkpad, Flip, and Scale-up, maintaining high ASR where BadNets fails completely. Additional analysis shows the patch trigger achieves a mean ASR of 0.95 across 200 triggers, and the L∞ (ϵ=16/255) trigger achieves 0.86. An attacker simulation using stable-diffusion-xl-base-1.0 to synthesize reference images from arbitrary captions confirms the method's practical potency, with patch triggers achieving 89.15% normal ASR and 86.50% trigger-level ASR.

Improvements to AI systems:

1. Dynamic, instruction-following backdoor defense evaluation: Build a new red-team benchmark where backdoors are evaluated on unseen trigger-target pairs, not just fixed pairs. This forces defenses to detect the underlying trigger-as-instruction generalization mechanism, not memorized patterns. The improved AI system can proactively identify and neutralize any-to-any backdoors that adapt at inference time.

2. Feature-space steganography detection: Develop a defense that monitors the feature-space alignment between input perturbations and known trigger prototypes (e.g., using the same pre-trained feature extractor Φ). The improved system can flag inputs whose visual features unexpectedly match a semantic caption's embedding, catching norm-controlled perturbations (L∞ 8/255 or 16/255) that are imperceptible in pixel space but detectable in feature space.

3. Robustness-aware training against programmable attacks: Incorporate the heuristic poisoning strategy (diverse trigger-caption pairs) into adversarial training for VLMs, but in reverse—train the model to ignore the trigger-as-instruction rule by adding a regularization term that penalizes caption changes when a trigger-like patch is present. The improved VLM becomes resistant to any-to-any attacks without sacrificing clean performance on GQA/MME benchmarks.

4. Cross-modal consistency checking: Use the feature-space trigger steganography method T S(·) as a defensive tool—after generating a caption, re-encode it and check if it matches any hidden trigger features in the input image. The improved AI system can reject inputs where the image contains a steganographic trigger that encodes a different caption, preventing malicious manipulation of downstream decision-making.

5. Adaptive trigger synthesis for safe AI alignment: Repurpose the attacker's trigger-generation pipeline for positive control—e.g., embedding safety instructions as imperceptible visual triggers that steer VLMs toward safe outputs (e.g., refusing harmful prompts). The improved system can dynamically inject safety semantics via triggers without retraining, enabling real-time policy updates in deployed models.

6. Generalization-aware backdoor auditing: Use the paper's finding that vanilla triggers achieve 92% ASR on unseen pairs to build an auditing tool that tests whether a VLM has learned a generalizable backdoor rule. The improved system can run a small set of random trigger-caption pairs to estimate the model's vulnerability surface, flagging models that exhibit high any-to-any ASR for further inspection.

7. Stealthy trigger-robust image preprocessing: Implement a preprocessing layer that applies random geometric transforms (e.g., Shrinkpad, Flip, Scale-up) and feature-space denoising (e.g., projecting inputs onto a clean manifold using Φ) to break the steganographic trigger's feature alignment. The improved system maintains high utility on benign inputs while reducing the attack's ASR from 86-92% to near-zero, as shown by the paper's defense robustness results.

Existing vision-language model (VLM) backdoors are usually treated as static vulnerabilities: one-to-one and N-to-N attacks bind one or more triggers to a finite set of targets before victim training. This assumption substantially underestimates the threat. We show that a single poisoning phase can implant a programmable backdoor into a VLM, allowing an attacker to choose previously unseen target-caption semantics at inference time and synthesize corresponding stealthy triggers on demand. Unlike fixed-mapping attacks, the proposed any-to-any caption-control paradigm decouples post-training target selection from poisoning, enabling dynamic control of target captions without retraining the VLM. Our method has two components. First, a heuristic poisoning strategy exposes the model to diverse trigger-caption pairs, encouraging it to learn a general trigger-as-instruction rule rather than memorize a specific backdoor pattern. Second, a feature-space trigger steganography method maps any attacker-specified target caption to a stealthy visual trigger, implemented as either a norm-controlled perturbation or a non-semantic patch. Once inserted into arbitrary images, these triggers cause the poisoned VLM to generate outputs semantically aligned with the chosen target caption, even when the target was unseen during poisoning. Extensive experiments show that our attack achieves high any-to-any caption-control success rates, preserves clean model utility, and remains effective under several classical backdoor defenses.