Is Success All You Need? Investigating the Impact of Input Perturbations on VLA Behaviour in Tabletop Manipulation Tasks

Vision-Language-Action (VLA) models have achieved high task success rates on robot manipulation benchmarks, but this work proposes a benchmark-agnostic evaluation framework to measure behavioral robustness by characterizing how successful trajectories are executed under input perturbations. The core finding is that perturbations can alter the behavior of successful trajectories, a phenomenon not captured by Task Success Rate (TSR) alone.

How it works

The proposed methodology extends widely-used benchmarks like LIBERO and LIBERO-Plus to evaluate the behavioral robustness of state-of-the-art VLA models. The framework moves beyond binary task success by calculating metrics that characterize the nature and variability of successful task execution, complementing TSR. The evaluation involves analyzing model behavior under original, unperturbed tasks as a baseline and under various perturbation conditions.

The specific behavioural evaluation metrics include:

1. Duration (s): Calculated as elapsed simulator time between the first and final recorded control step using the simulator control frequency to isolate task completion time from inference latency.

2. Total gripper movement (m): The cumulative absolute change in gripper width for a symmetric two-finger gripper, capturing total open/close motion.

3. Cartesian jerk (m/s3): Computed as the third derivative of the Cartesian position, capturing how abruptly the end-effector motion changes and thus trajectory smoothness. Mean and P95 versions are calculated to capture average and upper-tail behavior, respectively.

4. Joint jerk (rad/s3): Captures the smoothness of movement of the arm as a whole.

5. Cartesian path length (m) and Joint path length (rad): These metrics capture how far the end-effectors move through physical space and how much the robot arm configuration changes over time, respectively.

Quantifying typical behaviour change

To quantify how successful trajectories change under perturbation, the researchers calculate a distribution of trajectory-level metric values for both baseline and perturbed conditions. For each task, metric, and perturbation condition, they compute the mean metric value across the distribution of successful trajectories (Equation 5). This yields a task-level percentage change in the mean metric value relative to the no-perturbation baseline. The overall effect of a perturbation across an entire task suite is summarized by reporting the median of these task-level percentage changes, ∆e m,p = mediank (∆k,m,p) (Equation 6). Statistical significance is assessed using a two-sided Mann-Whitney U test with Benjamini-Hochberg false discovery rate correction.

Quantifying behaviour variability

In addition to characterizing typical behavior change, the framework quantifies behavioral variability using the median absolute deviation (MAD). For each task, metric, and condition, MAD is calculated as the median absolute deviation of trajectory-level metric values (Equation 7). The percentage change in MAD relative to the baseline is then calculated as ∆MADk,m,p (Equation 8). Positive values indicate increased behavioral variability. The median of these task-level percentage changes across all tasks in a suite is reported to summarize variability effects.

Key findings on behaviour and TSR correlation

The study investigates three research questions: RQ1 (How does typical behavior change?), RQ2 (Are perturbation-induced behavioral changes reflected by changes in the TSR?), and RQ3 (Do perturbations alter the behavioral consistency of successful trajectories?).

Regarding typical behavior, the results indicate that successful trajectories are not behaviourally invariant to perturbations. For instance, notable examples from the Spatial suite showed a 24.1% (7/10 tasks significant) and 35.3% (10/10 tasks significant) median increase in mean Cartesian jerk under the camera perturbation for the π0.5 model. This sensitivity is model-dependent; The robot and camera perturbations appear to have the most significant effect on the behaviour metrics of π0.5 and VLANeXt models.

Concerning TSR correlation, a Spearman rank correlation analysis revealed moderate positive correlations between TSR degradation and total gripper movement (ρ=0.56), P95 Cartesian jerk (ρ=0.54), duration (ρ=0.52), mean Cartesian jerk (ρ=0.43) and mean joint jerk (ρ = 0.4). This suggests that perturbation conditions associated with larger reductions in task success also tended to produce larger behavioral changes in successful trajectories, indicating that changes in task success do not fully capture changes in successful execution behaviour.

Key findings on behavioural consistency

The analysis of behavioral consistency shows an interesting pattern: the perturbation conditions which produced the largest shifts in typical successful behaviour also frequently increased behavioural variability (Table III). Notably, the camera perturbation led to an increase in MAD for every evaluated model and suite across duration, mean Cartesian jerk and gripper movement. Conversely, several perturbations led to reduced MAD when compared with the unperturbed baseline, such as background, language, lighting and object conditions. This suggests that perturbations might be narrowing the set of behaviours that enable success.

Based on the scientific paper, here are specific improvements that can be made to existing Vision-Language-Action (VLA) models and what those improved systems could achieve:

1. Improve model evaluation by moving beyond Task Success Rate (TSR) as the sole metric for robustness.

2. Implement a benchmark-agnostic framework that complements TSR with trajectory-level behavioral metrics, specifically focusing on motion smoothness, efficiency (duration), and gripper behavior (total movement).

3. Analyze how perturbations change the mean and variability of these specific behavioral metrics during successful task execution (RQ1).

4. Quantify the correlation between TSR degradation and changes in these behavioral metrics to determine if success is decoupled from good execution quality (RQ2).

5. Assess whether different models exhibit differential robustness—i.e., some models may maintain high TSR while their successful trajectories suffer substantial behavioral degradation under the same perturbation (RQ2).

6. Determine if perturbations alter the consistency of successful behaviors by calculating and analyzing the Median Absolute Deviation (MAD) for these metrics, identifying which perturbations lead to more or less reliable execution strategies (RQ3).

This improved AI system would be able to:

1. Identify VLA models that achieve high task success rates but execute those successes in a brittle or inconsistent manner when faced with real-world environmental variations (e.g., changes in lighting, camera angles, or robot state).

2. Provide a more comprehensive model card for VLA systems by reporting not just whether they succeed, but also the quality of their successful actions (e.g., smoothness and energy efficiency).

3. Diagnose precisely which types of input perturbations (camera changes vs. language instruction vs. lighting) cause specific degradation in robotic movement patterns during task completion.

4. Help developers understand the trade-off between achieving a high TSR and maintaining a predictable, smooth, and efficient execution profile for physical deployment in real-world scenarios.

5. Guide the fine-tuning process by suggesting methods to improve trajectory consistency (reduce MAD) under challenging conditions to ensure reliable operation in variable environments.