HERO: Learning Humanoid End-Effector Control for Visual Whole-Body Open-Vocabulary Object Grasping

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa.

Dev: And I'm Dev, here with Taro.

Rosa: Today's paper: "HERO: Learning Humanoid End-Effector Control for Visual Whole-Body Open-Vocabulary Object Grasping".

Dev: The gist:

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: So we're looking at this paper called "HERO: Learning Humanoid End-Effector Control for Visual Whole-Body Open-Vocabulary Object Grasping," and the authors are Runpei Dong, Ziyan Li, Arjun Gupta, and Xialin He. What's the main point here in plain language?

Dev: It’s about giving humanoids the ability to pick up stuff they haven't seen before in any room. The title says it combines visual whole-body open-vocabulary object grasping with an accurate end-effector tracking policy, and that sounds pretty ambitious for real life application.

Taro: It suggests these systems can handle reaching across tables, rotating the torso for objects on the side, or squatting for low tables while keeping things balanced. That level of dexterity in novel scenes is what they are aiming for with this work.

Rosa: Exactly, it’s moving beyond just picking up known items to actually reaching into an environment and figuring out how to grasp something new based on a simple text command like "pick up the orange mug." It tackles that fundamental problem of generalizable scene understanding and precise physical control all at once.

Dev: It suggests they are breaking the monolithic end-to-end learning approach that most people are using right now because that makes scaling way too hard for real-world use. Instead, they separate the planning from the execution, which sounds like a much more manageable architecture for building robust systems.

Taro: Separating those parts means if the vision model gets confused about an object, you only have to worry about correcting the grasp prediction part, not re-learning how to move your whole body from scratch every time.

The paper's summary: Rosa: So what is the actual framework they propose? They use open-vocabulary large vision models like Grounding DINO and SAM to first segment out the object of interest from a picture, and then they use an AnyGrasp model to predict parallel jaw grasps for that object.

Dev: That part handles the perception and grasping strategy—figuring out what to grab and how to grab it—and then they retarget that grasp prediction onto a specific hand, like a Dex-three hand <ref:2602.16705#pg3>. They are using this visual information to decide the action plan.

Taro: Then they take that whole-body reference motion, which comes from something called cuRobo, and use their proposed whole-body end-effector tracker to guide the robot arm toward that predicted grasp before it actually picks up the object.

Rosa: That tracking policy is where the core technical innovation lies, which they call HERO. It’s designed to convey the robot arm to that specific predicted grasp using a combination of classical robotics and machine learning techniques.

Dev: The paper claims this HERO policy reduces end-effector tracking error down to two point four four centimeters, which they say is five point five times better than the strongest prior method they looked at. That’s a very concrete number for control accuracy in the real world.

Taro: That tracking policy sounds like it’s not just trying to hit a target blindly; it involves sending motion planner output, current and target joint angles, and also the current and target end-effector position as inputs to make the control more informed.

Rosa: It suggests that by doing this decomposition—separating identifying which object to pick from the actual movement of the robot arm—they inherit a lot of general visual knowledge from those pre-trained models while building very strong control capabilities for training their tracking policy in simulation.

The paper's improvements: Dev: The authors point out several things they did to make this system better, and I want to run through those improvements now. First, they used two learned neural forward models to fix problems with the analytical kinematics that often causes inaccuracies in robot modeling.

Taro: They learn a residual leg odometry model, which learns the gap between what MOCAP odometry says about the legs and what their analytical forward kinematics calculates for the legs. This correction helps improve how they estimate where their feet are.

Rosa: And they also learn a residual end-effector pose model, which learns that specific gap between what their analytical FK predicts for the end-effector and where it actually ends up, using MOCAP data as ground truth. They train both models with mean squared error losses on translations and rotations to make sure they are accurate.

Dev: On top of those learned models, they have this closed-loop replanning system that periodically replans from the current state every three hundred steps, or six seconds, using cuRobo and an odometry-corrected goal. This is meant to compensate for drift during long operations.

Taro: And they also have a specific mechanism to fix steady-state end-effector tracking offset. They maintain a commanded grasp initialized to TEE and apply a damped translational update, where the grasp position gets nudged by negative beta times the end-effector position, only when that positional error is less than fifteen centimeters.

Rosa: So these improvements show they are tackling both modeling errors in the kinematics and operational drift during long tasks by adding these specific learning and replanning layers to their core tracking policy.

Conclusion: Dev: So to wrap this up, the paper presents a modular humanoid system that successfully separates open-vocabulary action planning from simulator-trained action controllers, which they show achieves ninety percent success on daily object grasping without needing large-scale real-world imitation.

Taro: The main lesson here seems to be that the task space state has to be really accurate; they found that the analytical forward kinematics has a one point three zero centimeter end-effector error, while their learned model reaches a zero point five zero centimeter error.

Rosa: And another big thing is how explicit error feedback during closed-loop execution really cuts down on real-world drift; their tracker reaches a two point five centimeter error in those tests. That’s pretty tight control when you're out there manipulating things without constant supervision.

Dev: It proves that having accurate task-space state is important, and the explicit feedback during closed-loop execution helps reduce those drifts that you see in practice.

Taro: And it shows that even with all these learning components, the system can be language sensitive under distractors in cluttered layouts, which suggests good resilience when things get messy.

Rosa: So "HERO: Learning Humanoid End-Effector Control for Visual Whole-Body Open-Vocabulary Object Grasping" gives us a way to build these systems by focusing on accurate state representation and using explicit feedback loops to keep the control stable. We’re ready to see what the next paper is about.