LIBERO-RECOVER: Beyond Task Success Towards Failure Recovery in Robotic Manipulation Models

Vision-Language-Action (VLA) and World Action (WAM) models have achieved remarkable performance in robotic manipulation, but success under ideal conditions does not imply real-world robustness because robots must recognize and recover from failures to continue tasks. This paper introduces LIBERO-Recover, a large-scale benchmark designed to evaluate the critical capability of failure recovery in embodied models by collecting real execution failures and constructing scenarios across four progressive difficulty levels.

The Gist

LIBERO-Recover is a large scale benchmark for failure recovery in robotic manipulation, shifting evaluation from Can the robot succeed? to Can the robot recover after failure?, by collecting real execution failures from SOTA embodied models and constructing 1,000+ scenarios across four recovery levels: (1) Action Retry, (2) Action Adaptation, (3) Object State Recovery, and (4) Environmental Recovery.

Problem Formulation

The paper contrasts conventional task execution with failure recovery. Conventional evaluation measures whether a policy can continuously execute the task from an initial state to a goal state. Failure recovery considers an execution that deviates from the intended progression and reaches an intermediate state where the original task goal is not satisfied, requiring subsequent interaction to restore necessary states before continuing toward the original goal. A failure state is defined as one where execution-induced changes invalidate the current plan, necessitating adaptation or recovery, while state changes that preserve task executability are treated as normal variations.

Failure Taxonomy and Difficulty Levels

The recoverable failures are organized into four progressive difficulty levels based on the required state reasoning and recovery effort:

1. L1 Action Retry: The failure does not materially alter the task configuration; recovery only requires reattempting the failed action.

2. L2 Action Adaptation: The failure causes a limited change, requiring adaptation of actions to the observed state rather than blindly repeating the failed action.

3. L3 Object State Recovery: The failure substantially alters a task-relevant object, requiring first restoring the object’s required state before resuming the original task.

4. L4 Environmental Recovery: The failure affects an object or state not directly manipulated by the original task but subsequently prevents execution, requiring reasoning about both task-relevant and task-irrelevant states and their interaction topology.

LIBERO-Recover Benchmark Construction

The benchmark is built upon LIBERO by collecting real execution failures from SOTA embodied models. The construction pipeline involves three stages:

1. Stage 1: Task Execution, where multiple embodied policies are deployed to generate a complete failure trajectory, yielding videos of the execution.

2. Stage 2: Failure Localization, utilizing Qwen3.5-27B-Instruct to analyze execution videos and temporally localize deviations into pre-failure, during-failure, and post-failure stages.

3. Stage 3: Failure Characterization, where Qwen3.5 is prompted to assess the post-failure consequences and determine the corresponding recovery type and difficulty level according to the taxonomy defined in Section 3.3. This process results in 2178 recovery scenarios derived from 130 subtasks across four task categories, covering four recovery levels and 16 evaluation dimensions.

Evaluation Metrics

To assess model performance, three core metrics are proposed:

(Recovery Success Rate (RSR))

This measures whether a policy can successfully complete the original task after entering a failure state: RSR = 1/N X N i=1 G(s i T, l) = 1. Higher RSR indicates stronger failure recovery capability.

(Recovery Degradation (RD))

This quantifies execution capability lost after a failure: RD = 1 − Mpost + ε / Mpre + ε. Lower RD indicates better retention of execution capability.

(Recovery Consistency (RC))

This measures the stability of recovery across different failure states within the same task: RC = 1 − Stdk(Rk). Higher RC indicates more consistent recovery across failure states.

Key Findings

Experiments reveal that Success in predefined scenarios does not guarantee robust generalization to real failures, showing performance drops over 50% when exposed to naturally occurring failures. Furthermore, models excel at local correction (L1/L2) but struggle with state recovery (L3/L4). WAM policies exhibit more stable recovery, achieving higher Recovery Consistency (RC) than conventional VLAs due to their explicit modeling of action-conditioned state transitions. Finally, Smaller action chunks improve recovery by enabling more frequent state feedback, suggesting that effective failure recovery requires fine-grained closed-loop control. However, recovery training does not effectively transfer to failures encountered during standard task execution, indicating a failure-recovery transfer gap. Temporal context also improves performance, as providing the task initial frame as temporal context consistently increases recovery success rates.

Here are the specific improvements that can be made to AI systems based on the LIBERO-Recover benchmark, and what those improved systems will be able to do:

1. The core improvement is shifting model evaluation from simple task completion success under ideal conditions (standard benchmarks) to robustness and resilience against real-world failures (LIBERO-Recover).

2. Improved AI systems will move beyond achieving near 100% success rates in simulation to demonstrating the capability to recognize, diagnose, and recover from unexpected physical disruptions like failed grasps, collisions, and unintended object movements during live operation.

Specifically:

3. Improved systems will exhibit superior performance across four distinct recovery levels:

4. Recognize and execute simple retries (L1), such as reattempting a failed grasp when the target remains unchanged.

5. Adapt their actions to limited state changes (L2), adjusting the next action based on observed, non-catastrophic deviations from the plan, rather than blindly repeating the failed action.

6. Perform complex object state recovery (L3), such as restoring a displaced object's pose before continuing manipulation, requiring structured reasoning about object states and spatial relations.

7. Execute sophisticated environmental recovery (L4), which involves reasoning about both task-relevant and irrelevant states, identifying environmental obstructions, and restoring necessary configurations to resume the original task.

Furthermore:

8. Improved systems will demonstrate greater stability in their recovery behavior (higher Recovery Consistency) compared to current models, suggesting a transition from purely imitation-based learning to world-model-based reasoning that explicitly predicts action consequences and state transitions.

9. The use of smaller action chunks in the policy execution loop will enable more frequent state feedback, allowing the system to track evolving post-failure states more effectively, leading to finer closed-loop control during recovery.

10. Systems trained on failure scenarios will exhibit a reduced failure-recovery transfer gap, meaning they can reliably recover from failures that occur during standard, unperturbed task execution (i.e., improving overall success rates in real-world deployment).

11. Incorporating temporal context (the initial frame) into the policy's decision-making process will allow systems to compare the current scene against the intended task configuration, significantly aiding in identifying state deviations caused by failures and inferring what has been disrupted.