Failing Forward: Adaptive Failure-Informed Learning for Vision-Language-Action Models

summary

Video file (mp4)

The gist

The gist: Adaptive Failure-Informed Learning (AFIL) is an end-to-end framework that leverages failure trajectories as adaptive negative guidance for diffusion and flow-based Vision-Language-Action

In short

Adaptive Failure-Informed Learning (AFIL) improves Vision-Language-Action (VLA) models by using failure trajectories as adaptive negative guidance during policy sampling. The framework employs a Dual Action Generator architecture that jointly learns from both successful and failed behaviors. This allows the model to steer action generation away from failure modes, significantly boosting task success rates and robustness compared to standard success-only learning.

Key concepts

Vision-Language-Action (VLA) Models
These are AI models designed for robotics that take visual input (what the robot sees), language input (text instructions), and output actions (what the robot does). They aim to allow robots to perform complex tasks by understanding both the world around them and high-level commands.
Dual Action Generator (DAG)
This is a specific architecture where two separate generators are trained: one for successful actions (Gsucc) and one for failed actions (Gfail). Both share the same vision-language backbone. This setup allows the system to learn distinct patterns for what works and what doesn't, enabling failure awareness.
Adaptive Failure-Informed Sampling
During robot operation, AFIL uses recorded failure trajectories to guide action generation. It calculates a 'Failure-Informed (FI) score' that combines success predictions with a negative prompt from the failure generator. This score dynamically adjusts how strongly the model steers its actions away from known risky areas.
Adaptive Guidance Strength
The strength of the guidance applied during action generation is not fixed; it changes based on how different successful and failed behaviors are locally distinguishable. This modulation uses a cosine similarity strategy, increasing guidance when failures become more apparent in the current step.

Terminology used across episodes

This episode discusses

The paper

Failing Forward: Adaptive Failure-Informed Learning for Vision-Language-Action Models · Read on arXiv

United Imaging Intelligence Institute of Boston, MA, USA

Vision-language-action (VLA) models provide a promising paradigm for scalable robotic manipulation, yet their reliance on success-only behavioral cloning leaves them brittle; lacking corrective training signals, minor execution errors rapidly compound into unrecoverable, out-of-distribution failures. To address this limitation, we propose Adaptive Failure-Informed Learning (AFIL), an end-to-end framework that leverages failure trajectories as adaptive negative guidance for diffusion- and flow-based VLA policies. AFIL uses a pretrained VLA to generate failure rollouts online, avoiding the need for handcrafted failure-mode design or human-in-the-loop recovery. It then jointly trains Dual Action Generators (DAGs) for successful and failed behaviors while sharing a common vision-language backbone, enabling efficient failure-aware policy learning with limited parameter overhead. During sampling, the failure generator adaptively steers action generation away from failure-prone regions and toward more reliable success modes, with guidance strength determined by the per-diffusion-step distance between success and failure distributions. Experiments across in-domain and out-of-domain robotic manipulation tasks, covering both short- and long-horizon settings, show that AFIL consistently improves task success rates and robustness over existing VLA baselines, demonstrating its effectiveness, efficiency, and generality.

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa, and with me are Dev and Taro, guest researcher.

Dev: Today's paper: "Failing Forward: Adaptive Failure-Informed Learning for Vision-Language-Action Models".

Rosa: The gist:

Dev: First, who's behind it and why it matters.

Paper summary: Rosa: So, to recap, the paper "Failing Forward: Adaptive Failure-Informed Learning for Vision-Language-Action Models" proposes AFIL as an end-to-end framework <ref:2605.08434#pg1>

Dev: It claims that by using online failure rollouts as adaptive negative guidance, this approach can steer action generation away from failure prone regions and toward reliable success modes <ref:2605.08434#pg2>

Taro: The architecture they use is a Dual Action Generator, or DAG-VLA model that learns from both success and failure data simultaneously while sharing a common vision language backbone <ref:2605.08434#pg3>

Rosa: This setup lets them jointly train the success generator and the failure generator using both success and failure datasets, Ds and Df <ref:2605.08434#pg2>

Dev: The key mechanism is the Failure-Informed score which combines the success generator's prediction with a weighted negative prompt from the failure generator’s prediction <ref:2605.08434#pg1>

Rosa: And they modulate how strong that guidance is using a cosine similarity based strategy, where the guidance scale is proportional to the cosine distance between success and failure distributions <ref:2605.08434#pg1>

Taro: This modulation means the system gets stronger when it notices that failures are becoming more apparent locally <ref:2605.08434#pg2>

Dev: The authors show that this approach consistently improves success rates across diverse in-domain and out-of-domain manipulation tasks spanning both short and long horizon settings <ref:2605.08434#pg3>

Rosa: They showed gains of plus seven point four percent and plus twelve point zero percent over static methods on the long horizon settings in the Maniskill environment <ref:2605.08434#pg3>

Taro: On the LIBERO benchmark, they noted that DAG-VLA with AFIL consistently achieves the strongest performance among all methods <ref:2605.08434#pg3>

Dev: That performance improvement on LIBERO went from ninety-six point nine percent to ninety-eight point four percent on the most challenging long horizon setting <ref:2605.08434#pg3>

Rosa: So, to summarize this segment, AFIL is an end-to-end framework using failure trajectories as adaptive negative guidance for diffusion and flow based VLA policies <ref:2605.08434#pg1>

Taro: It’s a principled way to learn from failure data that avoids the need for handcrafted failure modes or human in the loop recovery <ref:2605.08434#pg1>

Dev: The paper emphasizes that this joint training of success and failure data with shared backbones enables efficient failure aware policy learning with limited parameter overhead <ref:2605.08434#pg1>

Rosa: This whole concept points toward a more scalable mechanism for robust action generation that learns from the distribution of both good and bad outcomes <ref:2605.08434#pg1>

Conclusion: Taro: So looking at the title "Failing Forward: Adaptive Failure-Informed Learning for Vision-Language-Action Models" what does that mean in practice?

Rosa: It means we’re moving past just teaching a robot to do things perfectly in simulation, and instead training it to handle the inevitable mistakes that happen when it's actually deployed <ref:2605.08434#pg1>

Dev: The authors are arguing that failure trajectories aren't just noise; they are valuable information for improving the policy’s robustness <ref:2605.08434#pg2>

Taro: They show that by using adaptive negative guidance based on the distance between success and failure distributions, we can dynamically adjust how much we care about failure signals <ref:2605.08434#pg1>

Rosa: This suggests a future where VLA models are inherently better at generalizing to real world deployment settings where errors and recovery demands are unavoidable <ref:2605.08434#pg1>

Dev: It moves the focus from just achieving perfect success in ideal scenarios to actually building systems that can manage imperfect execution dynamically <ref:2605.08434#pg1>

Taro: It's about making the AI more resilient when it encounters situations that are outside what it saw during training <ref:2605.08434#pg2>

Rosa: That’s the big picture, showing how modeling failure distributions can provide an efficient and scalable mechanism for robust action generation <ref:2605.08434#pg1>

More episodes

← Home