MoCaP: Mobile Code-as-Policies for Zero-Shot Mobile Manipulation from Passive Human Videos

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "MoCaP: Mobile Code-as-Policies for Zero-Shot Mobile Manipulation from Passive Human Videos".

Dev: The gist The MoCaP framework turns passive human video into guidance for an end-to-end mobile manipulation policy by extracting spatial memory and cues of how people prepare to interact with…

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: So we've talked about how MoCaP works and what those initial results look like, and now let's talk a bit more about the specific structure of this paper, focusing on what they call its improvements.

Dev: They focus on five main improvements in the MoCaP: first, they have a unified Code-as-Policy system that coordinates navigation and manipulation without needing to predefine those two stages separately.

Taro: That seems like a big step because it means the system can revise actions from both planning domains at the same time based on feedback from live geometry, which is crucial for real-time control.

Rosa: Right, so if it tries to move and bumps something, it can immediately adjust its plan for both where to go next and how to grab things simultaneously.

Dev: Second, they enhance object localization via this fine-grained spatial memory that extracts object relations, landmarks, and place connectivity from the videos. They claim this increases navigation success from seventy percent up to one hundred percent >

Taro: That improved localization means the robot can search much more effectively because it understands the context of where objects are in relation to each other across different places in those recordings.

Rosa: And third, they have human-guided base placement refinement, which uses that human approach prior to guide docking by inferring the direction from how a person positioned themselves before manipulation >

Dev: That's interesting because it turns arrivals into graspable configurations using only video data accumulated from everyday activity. It’s using behavior to solve a geometric problem.

Taro: It seems like they are turning the preparation phase of human interaction into an explicit guidance signal for the robot's approach, which is something that was missing in prior work >

Rosa: And fourth, they have closed-loop policy revision where the code generation policy is revised based on execution feedback; failed actions interrupt the remaining program to allow for plan revision >

Dev: That's a safety feature. If the robot messes up a step, it doesn't just keep going blindly; it stops and regens its plan in real-time against live constraints.

Taro: And fifth, they have task-independent memory reuse, meaning the knowledge extracted from passive videos is not tied to one specific request but serves as a persistent guidance layer for any open-vocabulary instruction >

Rosa: So essentially, they've built a reusable library of human interaction knowledge that the robot can tap into for many different tasks without needing a new demonstration every time.

Dev: That reusability is what makes it flexible for open-vocabulary systems; it’s not just solving one problem, but setting up the robot to solve many problems >

Taro: So the structure of MoCaP seems designed to handle the messy reality of real-world interaction by separating how we store that knowledge from how we execute the control policy >

The paper's summary: Rosa: Let's go back to what this whole process means in practice. The paper summarizes MoCaP as an agentic framework that takes passive human video recordings and turns them into guidance for an end-to-end mobile manipulation policy.

Dev: Essentially, the system extracts spatial memory and cues about how people prepare to interact with objects, which then informs both the navigation part—how it searches—and the search for a manipulation-ready configuration.

Taro: So instead of just having pre-programmed paths or fixed goal points, the robot gets dynamic guidance based on what it learns from human preparation in those videos >

Rosa: That's right, and they use this guidance together with current observations to coordinate navigation and manipulation primitives using a code agent.

Dev: This code agent generates Python programs that invoke robot-grounded primitives, like lift grasped object or prepare for manipulation, which are then executed in a closed loop.

Taro: The key is that the execution feedback from these primitives informs the next step of code generation, which lets it adjust its plan against live geometry and kinematic constraints >

Rosa: It’s this continuous loop of guidance from human video memory and real-time execution feedback that allows it to solve motions under the robot’s own physical limitations.

Dev: The system is designed to be robust because it relies on this multimodal input—the learned spatial knowledge from video and the current visual observations—to coordinate everything needed for a task.

Taro: It moves beyond just simple perception; it uses human activity as a source of complementary information about how interaction is prepared and performed, which motivates using human behavior as more than just a record of object locations >

The paper's improvements: Rosa: We talked about the specific mechanisms, but let's focus on what these improvements actually mean for how we build these systems.

Dev: The first improvement is that they don't separate navigation and manipulation stages in control; they use this unified Code-as-Policy system to coordinate them flexibly through executable code >

Taro: That means the planning domain can simultaneously revise where to go and what action to take based on feedback, which is much more efficient than having a strict handover between two different control modules >

Rosa: And the second improvement is that they developed this fine-grained spatial memory that draws on everyday human activity recordings to extract object relations, landmarks, and place connectivity for subsequent robot search and navigation >

Dev: That means the system gains much better object localization because it understands how objects relate to each other spatially across different scenes in those videos >

Taro: And third is the human-guided base placement refinement where they use the human approach prior to guide docking, inferring direction from how a person positioned themselves before manipulation >

Rosa: So it means we can use that inferred approach direction to find a graspable configuration even when our initial setup is not ideal >

Dev: Fourth is the closed-loop policy revision mechanism, where execution feedback informs the next code generation step, ensuring continuous refinement of navigation and manipulation primitives against live geometry >

Taro: And fifth is the task-independent memory reuse, meaning that knowledge extracted from passive videos isn't specific to one request but serves as a persistent guidance layer for any open-vocabulary instruction >

Rosa: So overall these improvements show that we can extract really rich, reusable spatial and interactive knowledge from everyday videos to guide complex, end-to-end mobile manipulation policies >

Conclusion: Dev: So to wrap up the discussion on MoCaP, it seems the main implication is that passive human video can provide a powerful source of experience for robots without needing specific demonstrations for every task.

Rosa: That's right, and by extracting spatial memory and approach cues, MoCaP provides guidance for both object search and the approach to manipulation >

Taro: It moves beyond just locating things to understanding how interaction is prepared, which is a richer form of knowledge that helps robots operate in unstructured environments >

Dev: The system achieves eighty percent success on two mobile manipulation tasks in an indoor environment, showing it can actually perform these complex actions where existing methods struggle >

Rosa: And the navigation success jumped from seventy percent to one hundred percent, which shows the memory really helps with searching beyond just what’s immediately visible >

Taro: It suggests that this kind of experience can fundamentally change how we think about learning for mobile manipulation because the robot isn't just reacting; it's guided by learned human context >

Dev: We need to keep in mind the limitations, though, like the evaluation being limited to one environment and one video recording, and also that local base search depends heavily on visible geometry and a grasp model >

Rosa: And manipulation is currently limited to grasping and lifting; it doesn't do sustained contact or precise placement yet >

Taro: The paper flags that the system design itself constrains performance because a fixed startup summary can omit relevant objects or become stale, and local base search only depends on visible geometry >

Dev: So the final thought is that while MoCaP shows promise for guiding object search and approach to manipulation from passive human videos, future work needs to evaluate these priors across multiple homes and recordings >

Rosa: That sounds like a good direction; exploring how this works in more diverse settings will really test how generalizable this spatial memory actually is >