Pixel2Catch: Multi-Agent Sim-to-Real Transfer for Agile Manipulation with a Single RGB Camera

summary

The gist

The gist: Pixel2Catch proposes an RGB-only robotic catching system that infers object motion from pixel-level features rather than explicit 3D position estimation, enabling successful sim-to-real

In short

Pixel2Catch proposes a robotic catching system using only a single RGB camera to infer object motion from pixel-level features instead of estimating 3D coordinates. It uses image segmentation to capture changes in an object's position and scale over time. By employing a multi-agent reinforcement learning framework, the system successfully transfers policies trained in simulation directly to the real robot without fine-tuning.

Key concepts

Pixel-level Feature Vector (zpixel): This vector captures essential visual information from an RGB image, including the object's center coordinates (cx, cy), width (w), and height (h). Crucially, it includes temporal differences between consecutive frames to represent motion dynamics. This allows the robot to understand how the object is moving in image space.
Segment Anything Model 2 (SAM 2): SAM 2 is used to robustly segment objects within the RGB image. It extracts pixel-level information that captures relative changes in the object's position and scale between frames. This feature extraction forms the basis for inferring object motion without needing explicit depth or metric coordinates.
Heterogeneous Multi-Agent Reinforcement Learning (MARL): The system uses a MARL framework to control a complex robot arm and hand by treating them as independent agents with distinct roles. The arm agent focuses on reaching the object, while the hand agent focuses on stable grasping. They learn cooperatively using role-specific rewards, leading to more stable and effective manipulation.
Centralized Training with Decentralized Execution (CTDE): This paradigm is used during training where all agents share information centrally to learn complex interactions. During execution in the real world, each agent acts independently based only on its local observations. This structure helps achieve stable learning for high-DoF systems.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Retrieval-Augmented Generation (RAG) — explained in 21 episodes
Sample Complexity — explained in 20 episodes
Flow Matching — explained in 19 episodes
Quantum Fisher Information (QFI) — explained in 19 episodes
Federated Learning — explained in 17 episodes

This episode discusses

The paper

Pixel2Catch: Multi-Agent Sim-to-Real Transfer for Agile Manipulation with a Single RGB Camera · Read on arXiv

Seongyong Kim, Junhyeon Cho, Kang-Won Lee, Soo-Chul Lim

Dongguk University

To catch a thrown object, a robot must be able to perceive the object's motion and generate control actions in a timely manner. Rather than explicitly estimating the object's 3D position, this work focuses on a novel approach that recognizes object motion using pixel-level visual information extracted from a single RGB image. Such visual cues capture changes in the object's position and scale, allowing the policy to reason about the object's motion. Furthermore, to achieve stable learning in a high-DoF system composed of a robot arm equipped with a multi-fingered hand, we design a heterogeneous multi-agent reinforcement learning framework that defines the arm and hand as independent agents with distinct roles. Each agent is trained cooperatively using role-specific observations and rewards, and the learned policies are successfully transferred from simulation to the real world.

DOI: 10.1109/LRA.2026.3703582

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "Pixel2Catch: Multi-Agent Sim-to-Real Transfer for Agile Manipulation with a Single RGB Camera".

Dev: Here's the gist of the paper. The gist: Pixel2Catch proposes an RGB-only robotic catching system that infers object motion from pixel-level features rather than explicit 3D position estimation, enabling successful sim-to-real transfer.

Dev: The system is inspired by human visual perception, where object motion is inferred from "pixel-level features in image space rather than metric 3D coordinates" <ref:2602.22733#pg2>.

Paper summary: Rosa: So, to start with Pixel2Catch: Multi-Agent Sim-to-Real Transfer for Agile Manipulation with a Single RGB Camera, what's the big idea here? It claims they can catch thrown objects using only an RGB camera without ever estimating the object's precise three dee position <ref:2602.22733#pg1>. Dev That's right. The authors are moving away from needing depth sensors or motion capture systems to get that metric three dee data <ref:2602.22733#pg1>. Rosa They say instead, you infer the object's motion just by looking at pixel-level features in the image space, which captures changes in the object’s position and scale over time. Dev It means they are using visual cues directly from a single RGB image to figure out how things are moving dynamically. Rosa And to make this work stably on a complex system with both an arm and a hand, they use this heterogeneous multi-agent reinforcement learning framework, treating the arm and the hand as independent agents with different jobs. Dev They train these agents cooperatively using role-specific observations and rewards so that each part learns its specific task well. Rosa The core claim is that because they do this training in simulation, they can transfer those learned policies directly to the real robot without needing any extra tuning or fine-tuning in the physical world.

Taro: So, if I'm driving and I see a ball fly toward me, I don't need a precise coordinate of where it is right now; I just see how fast it's moving across my vision. Rosa Exactly. This approach aims to capture that relative motion using pixel-level features like the object’s image space center coordinates, width, and height. Dev They define this feature vector as zpixel, which includes those spatial dimensions plus temporal differences between consecutive timesteps to capture the motion dynamics. Taro It sounds like they are letting the AI learn a visual language for catching by focusing on what's changing in the image itself rather than trying to solve a hard geometric problem.

Conclusion: Rosa: Looking at Pixel2Catch: Multi-Agent Sim-to-Real Transfer for Agile Manipulation with a Single RGB Camera, the authors are showing how you can build a system that works well in simulation and then actually performs in the real world without needing extra work on the robot. Dev The main contribution they highlight is this novel framework that represents object motion using pixel-level features extracted from an RGB camera rather than explicit three dee position estimation <ref:2602.22733#pg1,object motion using pixel-level>. Rosa It simplifies things for robotic catching because it makes you rely only on visual motion patterns, which is much closer to how we catch things in the real world. Dev And they achieved this by decomposing the system into a robot arm and a multi-fingered hand, each trained with its own set of observations and rewards. Rosa It really boils down to proving that you can get agile and stable catching using only RGB visual input if you set up your learning framework correctly in simulation.

Taro: What this means for us is that we don't have to wait for perfect, expensive sensor data just to train a robot arm. Rosa It shows that the motion cues within an image are rich enough for the policy to reason about the object's movement effectively. Dev The implication is that we can build complex manipulation systems by focusing on how things look and move visually, instead of being completely bottlenecked by perfect three dee sensing <ref:2602.22733#pg1>.

Taro: But they also pointed out a limitation, which is important for us to consider later. They mention that performance degradation in real-world settings for baseline methods is due to differences in object trajectory distributions between simulation and the real world. Rosa So, while they transferred it successfully, the performance might still drop if the object moves in a way that's totally outside what was randomized during training.

Dev: That makes sense from a control standpoint. If the real world throws an object with a completely different trajectory distribution than what they saw in simulation, even with this good visual feature extraction, the learned policy might struggle to adapt gracefully.

Rosa: So, the Pixel2Catch approach is powerful because it skips that explicit three dee estimation step entirely and relies on learning from those image space changes <ref:2602.22733#pg1>.

Dev: It's a solid demonstration of how multi-agent reinforcement learning can handle high degrees of freedom by separating the roles of reaching and grasping in a way that makes training more stable.

More episodes

← Home