Recova: Agent-Guided Failure Recovery for Autonomous Robotic Manipulation

Manipulation failures can leave scenes from which a task policy cannot recover, and Recova presents an agent-guided framework that jointly develops task execution and recovery in a reconstructed digital twin, then verifies and refines both through real-world experience.

The gist

Recova is an agent-guided framework that develops task execution and scene recovery in a reconstructed digital twin, then continually improves both through real-world experience.

How it works

Recova decouples task execution from scene recovery and connects the two through an agent that coordinates simulation, deployment, and learning. This separation trains each capability on the data suited to its role and lets recovery skills accumulate independently of any single task. The framework involves several coordinated components:

1. A coding agent builds a digital twin of the workstation from real-robot recordings (camera observations, robot trajectories, and calibration) using a MuJoCo scene.

2. The agent explores failures in the twin; upon failure, it diagnoses the issue and develops corrective behaviors in two forms: a program for a skill library and recovery rollouts that form the recovery dataset.

3. Successful task rollouts initialize separate task and recovery policies, while programs capture reusable recovery strategies.

4. During real-world deployment, a vision-language monitor tracks progress through four steps: Monitor (checking completion), Detect (deciding if intervention is needed and naming a recovery instruction z), Decide (triggering autonomous or human intervention based on the situation), and Verify (checking scene restoration).

Learning Across DAgger Rounds

Recova refines the task and recovery policies through repeated rounds of DAgger-style data collection. During each round, both policies remain fixed while the agent collects trajectories from robot execution and human demonstrations. These collected trajectories are divided into four groups: Sk (Success), Ek (Expand, containing human task demonstrations), Rk (Recover, containing human recovery demonstrations labeled with a recovery instruction z), and Ak (Autonomous recoveries that pass the scene-restoration check). At the end of each round, successful task rollouts and human task demonstrations are added to the task dataset, and human recovery demonstrations and verified autonomous recoveries are added to the recovery dataset. Each policy is then fine-tuned on its updated dataset using its standard supervised objective.

Real-Robot Rollout Loop

The real-robot rollout loop closes the gap between simulation and reality. The task policy runs autonomously, Recova intervenes only when needed, and the resulting experience becomes training data that further improves both policies. The monitor implements four steps:

1. Monitor: A requirement query converts the task instruction into an explicit completion condition (Cg).

2. Detect: An intervention query compares recent frames with the initial scene to decide whether to intervene, reporting whether the scene is intact and naming a recovery instruction z if needed.

3. Decide: If stalled progress occurs in an intact scene, a human operator provides a task demonstration; otherwise, the recovery policy executes z if available or requests human help.

4. Verify: A restoration query uses initial images to check whether the scene is workable again, allowing the robot to resume execution after autonomous recovery.

Policy Refinement and Skill Expansion

The framework allows for specialized learning by keeping task and recovery datasets separate, enabling each policy to learn its distinct role: completing the task or restoring a scene from which task execution can resume. The recovery skill set also expands as the robot encounters new failures; when no known instruction applies, the intervention query proposes a new instruction based on human demonstration. This demonstration provides the first training example for the new skill, and it is added to K and guides the development of recovery programs in the digital twin. After training, this instruction is added to the recovery policy’s registry, making it available for autonomous execution in later rounds.

Performance Results

Recova achieves superior performance across benchmarks. On simulation benchmarks, Recova exceeds its strongest baseline by 7.1 percentage points on LIBERO-Pro and 26.9 points on MolmoSpaces. On real-robot tasks, DAgger fine-tuning more than triples the base task policy’s mean success (from 23.8% to 77.5%), and enabling recovery skills at deployment raises it further to 87.5%, the best result on every task across four real-robot settings. Across four collection rounds on one task, observed human intervention falls from 87.5% to 0%. This demonstrates how agent-guided recovery turns failures into reusable capabilities, improving robustness while progressively reducing human intervention.

Conclusion

Recova establishes scene recovery as a learnable capability that complements task execution and shows that agents can turn failures into reusable skills through an agent-guided real-to-sim-to-real framework.

Here are the specific improvements to AI systems based on the Recova framework, and what those improved systems can achieve:

) 1. Development of a Joint Task-and-Recovery Policy Architecture:

The core improvement is decoupling task execution from scene recovery by training them as separate policies, coordinated by an agent.

The improved system will feature a dedicated Recovery Policy trained specifically on restorative actions (e.g., pushing a stuck ring down) and a Task Policy trained on achieving the primary goal (e.g., stacking rings). This specialization allows each policy to become highly proficient in its domain, leading to higher mean success rates compared to monolithic VLA or code-as-policy models.

) 2. Agent-Guided Digital Twin Exploration (Sim-to-Real Bridging):

The system will incorporate a coding agent that uses a reconstructed digital twin (MuJoCo scene based on real camera/trajectory data) to proactively explore failure modes safely.

The improved system can autonomously discover novel, complex recovery skills in simulation without risking physical hardware damage or requiring frequent manual resets. The agent generates recovery rollouts and seeds a reusable skill library (like the set of learned recovery programs) that is ready for deployment.

) 3. A Robust, Parallel Deployment-and-Learning Loop:

The system will operate on a parallel DAgger data collection harness across multiple real-robot stations, coordinating the policies' refinement concurrently.

The improved system can scale data collection significantly by utilizing multiple physical robots simultaneously. This parallel loop allows the Task Policy and Recovery Policy to learn from diverse failure scenarios concurrently, leading to faster convergence and more generalized recovery skills that are robust across different task variations (e.g., across six LIBERO-Pro settings).

) 4. Dynamic Human-in-the-Loop Intervention:

The system will employ a sophisticated Vision-Language Monitor that intelligently decides when autonomous recovery fails or is insufficient, triggering targeted human intervention only when necessary.

The improved system can achieve near-zero human intervention rates on mastered tasks (falling to 0% in the study). When failure occurs, the monitor doesn't just call for help; it diagnoses the exact failure mode and proposes a specific recovery instruction from a learned registry, ensuring that human demonstrations are targeted and immediately contribute to expanding the autonomous skill set rather than just fixing a single mistake.

) 5. Scalable Skill Library Management:

The system will maintain a comprehensive, version-controlled code-as-policy library for recovery skills, allowing learned programs to be reused across different tasks.

The improved system can achieve true reusability of corrective behaviors. A skill like Realign a ring on the peg rim developed for one task can be instantly invoked by any other task that encounters that specific failure pattern, drastically reducing the need to relearn solutions for repetitive problems.

) 6. Continuous Skill Expansion Guided by Failure Feedback:

The system will incorporate a mechanism where newly encountered failures automatically seed the development of new recovery skills and guide human demonstrations.

The improved system gains resilience against unseen failure modes. When it encounters a configuration it doesn't know how to handle, the agent proposes a new recovery instruction, which is then refined by human input into a permanent skill for that policy, ensuring the robot’s recovery capabilities grow continuously throughout its operational life.