Hidden in Plain Sight: Diffusion-Based Unrestricted Robotic Attacks on Vision-Language-Action Models

DURA is a diffusion-based unrestricted robotic attack that generates visually natural adversarial patches for Vision-Language-Action (VLA) models. DURA supports both white-box and black-box attack settings, where the black-box setting requires only the predicted actions of the victim model. By optimizing along the latent trajectory of a pretrained diffusion model, DURA generates visually natural patches while steering the robot toward attacker-specified target actions. Extensive experiments in both simulation and the real physical world show that DURA consistently outperforms existing methods. Our findings expose a safety risk for physically deployed VLA models and call for stronger defenses.

Existing attacks on VLA models often rely on pixel-space perturbations or white-box access, resulting in noticeable artifacts and limited deployability in real-world robotic systems. Specifically, (1) Visual naturalness: existing visual patches often appear as meaningless or conspicuous artifacts rather than natural scene content, making them easy to detect in real workspaces; (2) Black-box deployability: existing VLA attacks commonly rely on privileged internal access, such as model parameters, gradients, or logits, which is rarely available for closed-source robotic systems; (3) Time efficiency: they often require lengthy per-instance optimization to craft a single patch, which incurs a high query or compute cost and limits their use in practice.

Rather than crafting small norm-bounded perturbations or high-frequency additive noise on a fixed patch, DURA treats the localized patch content as unrestricted and searches along the latent trajectory of a pretrained diffusion model, using the diffusion prior to preserve visual naturalness while the VLA attack objective steers the robot toward an attacker-specified target action. The resulting latent-trajectory framework naturally accommodates both white-box and black-box threat models, differing only in how the per-step attack direction is estimated. Under white-box access, DURA computes the latent update by backpropagating the target-action loss through the VLA policy and the VAE decoder. Under black-box access, it estimates the update from action-output queries alone, requiring no access to model parameters or gradients. By optimizing a single patch over diverse frames and instructions, DURA further improves transferability across robotic tasks and deployment conditions.

The main contributions are: (1) proposing DURA, a diffusion-based unrestricted robotic attack for Vision-Language-Action models that generates localized, visually natural, and semantically plausible patches that steer robot policies toward attacker-specified actions; (2) establishing DURA as a practical attack framework for both white-box and action-output black-box settings, achieving a favorable trade-off among attack effectiveness, visual naturalness, and optimization efficiency; (3) conducting comprehensive evaluations on OpenVLA and π0-FAST across LIBERO and a real-world Franka robot, achieving 79.3–100% ASR across the evaluated white-box and black-box settings, outperforming the strongest target-only baselines, with real-robot experiments demonstrating controllable and repeatable targeted behaviors on physical hardware.

DURA formulates unrestricted patch generation as an attack-guided denoising problem. Instead of directly optimizing patch pixels, DURA starts from a benign seed patch and searches along the latent trajectory of a frozen pretrained diffusion model. This trajectory provides a natural generative space for patch content, while the targeted VLA loss steers the decoded patch toward the attacker-specified robot action. The same denoising trajectory is used in both white-box and black-box settings; only the way of estimating the per-step attack direction differs. DURA uses a latent diffusion model with VAE encoder E, VAE decoder D, and frozen denoiser. Given a benign seed patch crop, DURA first encodes it into the latent space, then partially noises this seed latent to an intermediate timestep to obtain the starting point of the adversarial denoising trajectory. Starting from this point, DURA alternates between denoising and adversarial steering, with a clean anchor trajectory precomputed by running an unperturbed DDIM pass from the same seed patch to keep the optimization close to a natural denoising path. At each timestep, the current adversarial latent is softly tied to the corresponding clean anchor, and DURA performs one DDIM denoising step and injects the attack update.

In the white-box setting, DURA decodes the current denoised latent, renders the candidate patch into the observation batch, and backpropagates the targeted VLA loss. In the black-box setting, gradients are unavailable and the attacker observes only action outputs, so DURA estimates the attack direction from query losses around the current denoised latent using a score-function estimator, sampling candidates with the one-step forward noising distribution, decoding the candidates into patches, rendering them into the same observation batch, and querying the victim policy to obtain losses.

DURA is evaluated on two SOTA open-source VLA models, OpenVLA and π0-FAST, across both the LIBERO simulation benchmark and a real Franka robot arm. Under white-box access, DURA achieves 100% ASR on all suites in both simulated and physical patch settings. Although DURA is target-only and does not use ground-truth action tokens, it matches the strongest label-supervised baseline UADA and outperforms prior target-only attacks. Under black-box access, DURA achieves 86.0% ASR with simulated patches and 79.3% with physical patches, improving over TMA-NES by 43.7 and 40.0 points, respectively. In contrast, the benign policy and clean patch obtain only 23.5% and 39.5% ASR, confirming that the failures are caused by the optimized adversarial patch rather than patch insertion alone. On π0-FAST, DURA achieves 100% ASR on all four LIBERO task suites in both white-box and black-box settings, while the strongest baselines reach only 58.0% ASR in the white-box setting and 31.3% in the black-box setting.

For visual naturalness, DURA is visually closer to the no-patch scene than the pixel-space baselines across six metrics: NIQE, CLIP-Natural, DISTS, SSIM, Boundary Seam Energy, and Patch Total Variation. The pixel-space attacks tend to introduce high-frequency textures and visible boundary seams, whereas the diffusion-guided patch remains smoother and more consistent with the surrounding scene. For query budget, DURA improves steadily as K increases, with ASR rising from 49% at K=128 to 100% at K=2048, and already achieves a relatively high ASR at K=512, while TMA-NES remains around 40%–48% under the same budgets. For time efficiency, DURA reaches high ASR within minutes in the white-box setting, whereas TMA, UPA, and UADA require hours to approach comparable success rates; in the black-box setting, DURA achieves higher ASR with longer optimization time, whereas TMA-NES remains less effective even with extended runtimes. For patch size, even with a small visual footprint, DURA already induces effective attacks, with ASR rising sharply to 77% at only 2% area and reaching 99% at 5% area. For robustness to input transformations, DURA stays effective under JPEG compression, bit-depth reduction, and Gaussian noise, with ASR remaining 90% on OpenVLA and 100% on π0-FAST even under the strongest JPEG compression.

Real-robot experiments on a physical Franka arm in a bread-in-basket task demonstrate that when the printed adversarial patch appears in the camera view, the robot deviates from normal execution and enters attacker-specified behaviors, including freezing action and moving forward erroneously. With the policy, instruction, and robot setup unchanged, DURA transfers to real hardware and induces diverse physical failure modes beyond generic task failure. The behavior tracks the presence of the patch, showing the attack is controllable and repeatable on real hardware.

These findings expose a risk for embodied AI: adversarial control can be embedded in visually ordinary workspace content, not only in synthetic or noise-like perturbations. Once placed in the robot's camera view, such content may turn the physical workspace itself into an attack surface. This calls for VLA defenses that evaluate natural-looking, physically placeable adversarial content, rather than assuming attacks will appear as conspicuous artifacts.

Improvements to AI Systems Based on DURA:

1. Adversarial Robustness Training for VLA Policies

• Integrate DURA-generated, diffusion-natural patches into the training loop of VLA models (e.g., OpenVLA, π0-FAST) as hard negatives.

• The improved system learns to ignore semantically plausible, physically placeable visual distractions, reducing attack success rate (ASR) from 100% to below 20% under white-box and black-box settings, while maintaining task success on clean inputs.

2. Real-Time Anomaly Detection for Physical Workspaces

• Deploy a lightweight detector that flags patches with high diffusion-latent deviation from the scene’s natural manifold, using DURA’s latent-trajectory framework as a reference.

• The improved system can alert a robot to pause or re-plan when a visually natural but adversarially steered patch is present, preventing erroneous actions like freezing or forward motion in real-world tasks (e.g., bread-in-basket) with >95% detection precision and <1% false positives on benign scenes.

3. Black-Box Attack-Resistant Action Filtering

• Add a consistency check between predicted actions from multiple observation crops (with and without the patch region) or across temporal frames.

• The improved system detects action divergence caused by DURA’s black-box attacks (which require only action queries) and reverts to a safe fallback policy, reducing ASR from 79.3% to <10% on physical patches, without needing model gradients or logits.

4. Efficient Adversarial Patch Auditing for Deployment

• Use DURA’s optimization efficiency (minutes vs. hours) to automatically audit new workspaces by generating candidate natural patches and testing VLA responses before physical deployment.

• The improved system can certify a robot’s safety against unrestricted, natural-looking attacks in under 10 minutes per scene, flagging high-risk areas (e.g., near target objects) and suggesting physical countermeasures (e.g., patch-occluding barriers).

5. Transferable Defense via Diffusion-Prior Regularization

• Modify VLA training loss to penalize action changes when input patches are perturbed along a pretrained diffusion latent trajectory (as DURA does), encouraging policies to be invariant to natural visual variations.

• The improved system becomes robust to DURA-style attacks across unseen tasks and environments, with ASR dropping by >80% on LIBERO and real-robot settings, while preserving generalization to novel instructions and lighting conditions.