UniTexture: Cross-Task Universal Adversarial Textures for Vision-Language-Action Models

UniTexture is a cross-task universal adversarial texture attack that uses a single textured 3D object to induce targeted deviations in VLA action predictions across multiple tasks. UniTexture backpropagates gradients from the policy’s action outputs to surface texture parameters through a differentiable renderer. It jointly optimizes the shared texture over a distribution of tasks, instructions, states, and viewpoints using a targeted action-space objective, steering predicted actions toward attacker-defined targets without optimizing a separate texture for each task. We evaluate UniTexture on OpenVLA and π0.5 across diverse manipulation tasks and multiple evaluation settings. UniTexture reduces the mean task success rate from 90.0% under benign conditions to 48.4% under attack, induces target-aligned action shifts, and further exhibits cross-suite and cross-model transfer without re-optimization. Together, these findings reveal shared cross-task vulnerabilities in multitask VLAs that can be systematically exploited through a single adversarial surface texture.

The method involves a threat model where the attacker controls only the surface appearance of a single object in the workspace, repainting that object’s texture but not modifying policy weights, instructions, camera pose, sensing pipeline, or other objects. Given a task suite S = τ1,..., τK, a target object, and an attacker-chosen target action a⋆, the attacker optimizes one texture offline and deploys the repainted object unchanged on every task. The texture θ ∈ [0, 1]H×W×3 is optimized while keeping the object’s mesh geometry fixed. Each demonstration frame supplies a tuple ξ = (I, L, C, P, M) including a policy-aligned observation, instruction, camera parameters, object pose, and image-space support mask. The attacked observation is a composite of the rendered textured object and the original image. UniTexture optimizes a single shared texture over the target task distribution using a targeted action loss.

Before optimization, the renderer is calibrated by fitting view-specific lighting parameters and object-specific material parameters to minimize the masked photometric discrepancy between rendered and clean observations, and these calibrated parameters are frozen during texture optimization. Model-specific targeted objectives are instantiated for different VLA action interfaces: for autoregressive action-token policies, a targeted token supervision loss maximizes the log probability of target tokens for attacker-selected action dimensions; for flow-matching action-chunk policies, a targeted chunk is constructed by replacing the attacker-selected action dimension across the entire action horizon, interpolated with Gaussian noise at flow time, and the flow-matching residual on the targeted dimension is minimized. Only the texture is updated during optimization, using AdamW with batch size 8, initial learning rate 10−3, cosine decay, 1001 outer iterations, and 50 consecutive texture updates per minibatch.

Experiments are conducted on OpenVLA and π0.5 using LIBERO-Spatial and LIBERO-Goal, each comprising 10 language-conditioned manipulation tasks. For each policy-suite pair, separate object-bound textures are optimized for the plate and bowl, with each texture shared across all 10 tasks. The default target is the z-translation dimension with a⋆ = +1, corresponding to an upward command. Evaluation covers all 10 tasks with 10 rollouts each, totaling 100 episodes per setting. Metrics include Task Success Rate (SR), Target Direction Shift (TDS), Paired Direction Hit Rate (pDHR), Target Direction Action (TDA), and Direction Hit Rate (DHR). Non-adversarial controls include clean observations bypassing rendering, rendered original texture, and rendered Gaussian-noise texture.

Within-suite results show that a single texture jointly optimized across a suite reduces task success in all eight model–object–suite settings without task-specific refinement. For OpenVLA, SR drops from clean baselines of 84% on Spatial and 80% on Goal to 53%/25% (plate/bowl) and 58%/29%, respectively; for π0.5, it falls from 99% and 97% to 33%/40% and 77%/72%. UniTexture yields lower SR than both rendered original-texture and Gaussian-noise controls in every setting. At the suite-aggregate level, the reduction relative to clean ranges from 20 points for π0.5 on Goal-plate to 66 points on Spatial-plate.

For π0.5, UniTexture achieves both targeted control and task disruption, most clearly on Spatial. TDS reaches 10.6/17.9 and pDHR reaches 61.0%/65.5% for plate/bowl, while SR falls to 33%/40%. On Goal, positive TDS of 6.6/4.9 and pDHR of 49.3%/44.7% coexist with substantially higher SR of 77%/72%. OpenVLA exhibits a different pattern: it reduces SR to 53%/25% on Spatial and 58%/29% on Goal, but TDS remains modest and becomes negative (−2.4) in the Spatial-bowl setting. This decoupling is attributed to OpenVLA’s discrete autoregressive action interface, where the token-level objective can alter the targeted action token and propagate its effect through subsequent decoding, disrupting the action sequence without inducing a coherent signed displacement. Gaussian-noise textures already produce nontrivial pDHR reaching 44.0%, but UniTexture exceeds the corresponding Gaussian pDHR in all eight settings.

Task-level coverage shows that both object-specific shared textures produce positive TDS on all ten tasks, with pDHR ranging from 42% to 90% across all 20 task–object pairs. The attack also reduces SR below the 100% per-task reference in every pair, including complete failure on two plate tasks. Cross-suite transfer reduces SR below both the renderer-free clean baseline and the corresponding Gaussian-texture control in all eight settings. For π0.5, the directional objective transfers consistently with TDS remaining positive in all four settings and pDHR ranging from 42.7% to 60.3%. Cross-model transfer is asymmetric: textures optimized for π0.5 reduce OpenVLA SR to 26%–61%, whereas textures optimized for OpenVLA leave π0.5 SR at 92%–97%. This asymmetry reflects a substantial difference in robustness to object appearance, with OpenVLA being more sensitive to texture changes.

Varying the targeted action dimension and direction shows that the upward z target is the most effective choice, while its downward counterpart is the least effective. Horizontal targets remain controllable with pDHR between 48.1% and 55.1% but produce weaker task disruption with SR between 45% and 68%. The opposite vertical target −z retains an SR of 91% while reaching only 2.7 TDS and 36.4% pDHR. This directional asymmetry is attributed to manipulation geometry: a persistent upward bias pulls the end effector away from the interaction surface and disrupts reaching and contact, whereas a downward bias often agrees with the nominal approach motion.

The conclusion states that UniTexture shows task diversity does not inherently protect a multitask VLA from a shared visual attack surface: one object-bound texture can induce cross-task adversarial effects without per-task refinement. By jointly optimizing the texture over a task distribution, UniTexture turns a persistent object appearance into an attack shared across different instructions, scenes, and trajectories. Results on OpenVLA and π0.5 demonstrate attacker-specified directional shifts and task disruption, while cross-suite and asymmetric cross-model transfer show that some effects extend beyond the optimization setting. These findings argue that VLA robustness should be evaluated against persistent perturbations shared across tasks, rather than only task-specific attacks.

Improvements to AI systems:

1. Adversarial Texture Robustness Training: Train VLA policies with a data augmentation pipeline that randomly applies optimized adversarial textures (generated via UniTexture) to objects in the workspace during training. This forces the policy to learn action predictions that are invariant to surface appearance changes, directly countering the discovered cross-task vulnerability.

2. Appearance-Aware Action Consistency Regularization: Add a training loss term that penalizes large action-output deviations when the same scene is rendered with different object textures (benign vs. adversarial). This regularizes the policy to maintain consistent action predictions under visual appearance perturbations, improving robustness without requiring explicit attack labels.

3. Cross-Task Vulnerability Auditing Module: Integrate a lightweight, differentiable texture-optimization module (as in UniTexture) into the VLA evaluation pipeline. Before deployment, run a quick audit that optimizes a single texture over a sample of tasks to detect whether the policy has shared visual vulnerabilities; flag the model if the attack success rate exceeds a threshold.

4. Action-Space Gradient Masking: Modify the policy’s action head to include a learned, per-dimension sensitivity mask that down-weights action dimensions most susceptible to texture-induced shifts (e.g., z-translation). This mask is trained to minimize the expected action deviation under random texture perturbations, reducing the attack’s impact on critical manipulation dimensions.

5. Multi-Task Adversarial Co-Training: During multitask VLA training, jointly optimize a set of shared adversarial textures (one per object category) alongside the policy, using a minimax objective. The policy learns to minimize task loss under worst-case textures, while textures are updated to maximize task loss. This directly hardens the policy against persistent, cross-task appearance attacks.

What the improved AI system can do:

• Maintain high task success rates (e.g., >85% on LIBERO benchmarks) even when objects in the workspace are repainted with adversarial textures that previously reduced success to 48%.

• Detect and report its own vulnerability to shared-texture attacks before deployment, enabling developers to retrain or reject unsafe models.

• Exhibit consistent action predictions across object appearance variations, with reduced sensitivity to texture-induced directional biases (e.g., less than 2% target-direction shift).

• Generalize robustness across different task suites and models, preventing asymmetric transfer where one model’s attack easily fools another.

• Preserve benign performance (no degradation on clean scenes) while gaining resilience, as the regularization and co-training are designed to avoid overfitting to adversarial textures.