SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation

arXiv:2602.16863 · cs.RO, cs.AI · Submitted 2026-02-18 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation".

Dev: SimToolReal introduces an object-centric reinforcement learning framework designed to enable zero-shot dexterous tool manipulation by training a single general-purpose policy in simulation and transferring it to novel real-world tools and…

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: So we're looking at this paper called SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation, and I'm curious what that title really means in plain terms. It sounds a bit dense with all the technical jargon.

Dev: It’s about training just one general policy in simulation that can then work on totally new tools and tasks without needing specific training for every single object or setup. That zero-shot aspect is what caught my attention because it cuts down a ton of engineering work we usually have to do for each new piece of hardware.

Taro: I see the core idea here is using an object-centric view, which frames tool manipulation as learning one policy to reach random goal poses for procedurally generated objects in simulation. It seems like they're abstracting away the complexity of tool-specific skills into a universal skill of goal reaching <ref:2602.16863#pg0>.

Rosa: Exactly, and I'm wondering if this abstraction holds up when we move things out of the controlled simulation environment and into the real world. Can this single policy actually handle the physical differences between, say, a thin marker versus a thick hammer?

Dev: That’s the million-dollar question for me; it hinges on how well that training objective translates. The paper suggests they are inducing core skills like initial grasp and reorientation by making the agent manipulate many different kinds of objects toward random poses <ref:2602.16863#pg1>.

Taro: And I think the real power is in how it handles things when the world throws a curveball. The paper focuses on what happens when the world misbehaves because they are looking at how the policy reacts to those random goal poses in simulation, which should give it some robustness <ref:2602.16863#pg1>.

Rosa: So, essentially, they're trying to learn the fundamental mechanics of manipulation—grasping and moving—in a way that makes them adaptable later. It sounds like they are trying to bypass the usual slow process of modeling every new tool from scratch <ref:2602.16863#pg0>.

Dev: Right, and for us engineers, it's promising because it shifts the heavy lifting away from per-object modeling and task-specific reward tuning which is usually a huge time sink. It suggests we can train something once and deploy it broadly <ref:2602.16863#pg1>.

The paper's summary: Rosa: Moving on, I want to get into the actual mechanics of how this system works, based on the summary they give us for SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation. What’s the main training objective they use?

Dev: The main idea is that in simulation, we train a goal-conditioned RL policy to manipulate a wide variety of procedurally generated objects toward randomly sampled goal poses <ref:2602.16863#pg1>. They use three specific reward terms: one for smoothness, one for grasping and lifting, and the main driver which is the goal-reaching term <ref:2602.16863#pg1>.

Taro: That goal-reaching term is what I find interesting because it's designed to be sparse; it only gives positive reinforcement when the agent successfully reaches a specific pose and then samples a new one, which should encourage learning the sequence of behaviors needed for tool use <ref:2602.16863#pg1>.

Rosa: That sparsity is smart, but I'm also interested in how they handle perception during inference when we actually deploy it on a real tool. How does the policy know what it's holding and where it needs to go?

Dev: The policy inputs are conditioned on several things: the current 6D tool pose, a coarse three dee grasp bounding box that encodes the intended graspable region, and an LSTM backbone which helps integrate interaction history to infer latent physical properties <ref:2602.16863#pg1>.

Taro: So it relies on that learned representation—the latent properties inferred by the LSTM—to handle things when we don't have perfect, direct observation of every physical detail <ref:2602.16863#pg1>.

Rosa: That seems like a sophisticated way to manage uncertainty during real-world deployment. It moves beyond just looking at raw sensor data and tries to build an internal model of the object <ref:2602.16863#pg1>.

Dev: And the whole pipeline for transferring this to reality uses vision foundation models like SAM three dee to generate meshes from human videos and then FoundationPose to extract sequences of 6D goal poses for deployment <ref:2602.16863#pg1>.

Taro: It sounds like they’re building a whole system around generating the necessary context—the object geometry, the grasp box, and the goal trajectory—before feeding that information into the RL policy during inference <ref:2602.16863#pg1>.

The paper's improvements: Rosa: Now let's discuss what they actually propose as improvements over previous methods. What is the main claim about how SimToolReal advances the state of this research?

Dev: The key improvement is moving away from methods that require substantial engineering effort for per-object modeling and task-specific reward tuning, which was a major bottleneck in prior sim-to-real RL approaches <ref:2602.16863#pg1>.

Taro: They claim this object-centric framework achieves strong generalization across diverse tools without requiring any object or task-specific training, which is a significant leap in terms of how broad the learned skills are <ref:2602.16863#pg1>.

Rosa: So, the improvement is fundamentally about achieving zero-shot deployment on novel tools and tasks just by mastering a universal manipulation skill in simulation <ref:2602.16863#pg0>. It simplifies the deployment pipeline significantly, doesn't it?

Dev: It does simplify things greatly because they rely on this single general-purpose policy trained in simulation, which is then transferred to real-world tools from DexToolBench <ref:2602.16863#pg1>. They aren't retraining the whole system for every new tool <ref:2602.16863#pg1>.

Taro: From an autonomy standpoint, this means the agent learns fundamental, transferable manipulation skills rather than just memorizing trajectories for a few specific tasks <ref:2602.16863#pg1>. That makes it much more flexible when things don't go exactly as planned <ref:2602.16863#pg1>.

Rosa: I see how that translates to real-world applicability, but I’m still worried about the gap between simulation and reality. How large is this generalization actually?

Dev: They show strong zero-shot generalization over one hundred twenty real-world rollouts across twenty-four tasks, twelve object instances, and six tool categories, outperforming prior methods using fixed grasps and motion retargeting by a factor of thirty-seven percent <ref:2602.16863#pg1>.

Taro: That comparison number suggests that the generalization capability is substantial enough to make this approach practically viable for deploying tools in varied environments, which is what we need for real autonomy <ref:2602.16863#pg1>.

Conclusion: Rosa: So, to wrap up our discussion on SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation, what are the most important implications we should be taking away from this work?

Dev: The main implication is that we can achieve dexterous tool manipulation with a single general policy trained in simulation, which bypasses the need for extensive per-object modeling and task-specific reward tuning <ref:2602.16863#pg1>.

Taro: From my side, I think the real implication is that we are learning more transferable skills that allow agents to handle unforeseen situations when the world misbehaves because of the way they are trained on random goal poses <ref:2602.16863#pg1>.

Rosa: And for me, it means that if this approach scales, we could dramatically expand the set of tasks a robot can perform just by changing the tools available to it <ref:2602.16863#pg0>.

Dev: We're looking at a system where the performance is measured against prior methods using fixed grasps and motion retargeting, showing an improvement of thirty-seven percent in generalization <ref:2602.16863#pg1>.

Taro: I think the future work should focus on making sure this single policy doesn't fail when the real-world interaction feedback is messy or incomplete, because that seems like a place where its current generalization might hit a wall <ref:2602.16863#pg1>.

Rosa: Exactly, and we’re ready to see what comes next in this area of research. We'll be sure to keep an eye on how this framework evolves <ref:2602.16863#pg0>.

Cornell University · Stanford University

cs.RO, cs.AI

Submitted: 2026-02-18

Updated: 2026-10-06

Comments: 23 pages, 16 figures, 3 tables. Project page: https://simtoolreal.github.io/

Code: https://github.com/Genesis-Embodied-AI/Genesis

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 89/100

The gist: SimToolReal introduces an object-centric reinforcement learning framework designed to enable zero-shot dexterous tool manipulation by training a single general-purpose policy in simulation and

Key concepts

Object-Centric Lens
This frames tool use as manipulating an object to a goal pose rather than a complex sequence of actions. By focusing on mastering the ability to reach any random pose with various objects, the framework learns fundamental skills like grasping and reorienting, which are then generalizable to any specific tool task.
Goal-Reaching Reward
The primary training reward drives the agent toward a target goal pose. This sparse reward system encourages the agent to learn how to successfully manipulate an object into a desired configuration. This objective is universal, meaning it trains the policy for manipulation skills applicable across many different tools and tasks.
Foundation Models in Transfer
The paper uses vision foundation models like SAM 3D to generate 3D meshes and grasp bounding boxes from videos. These models provide reliable object representations that condition the RL policy during real-world deployment, bridging the gap between simulated training data and novel physical tools.
Asymmetric Critic
This is an RL design choice where the critic (which evaluates the policy) has access to 'privileged states'—the ground-truth state from simulation. The actor (the policy being trained) only sees restricted observations. This setup helps the agent learn effectively despite partial observability in real-world scenarios.

Terminology

Summary

SimToolReal introduces an object-centric reinforcement learning framework designed to enable zero-shot dexterous tool manipulation by training a single general-purpose policy in simulation and transferring it to novel real-world tools and tasks. This approach is significant because it avoids the substantial engineering effort typically required for per-object modeling and task-specific reward tuning, achieving strong generalization across diverse tools without requiring object or task-specific training.

The gist: SimToolReal is a framework for training a single general-purpose, object-centric RL policy in simulation and transferring it to real-world tool use.

Core Concept and Problem Framing

The paper frames dexterous tool use as manipulating a tool through a sequence of goal poses. This object-centric lens reduces the complex problem of tool manipulation to learning a single goal-reaching RL policy that manipulates procedurally generated tools toward random goal poses. The key insight is that mastering the ability to manipulate objects to any random pose induces the core skills required for tool use: establishing an initial grasp, in-hand object reorientation, and maintaining stable contact. This abstraction allows a single goal-reaching controller to support diverse tool-use tasks without task-specific policy training or reward design.

Training in Simulation

The simulation environment is designed to induce the necessary skills through a universal objective. The robot learns by manipulating a wide variety of procedurally-generated objects toward randomly sampled goal poses. The training objective is defined by the reward function:

  1. A smoothness term, rsmooth, penalizes joint velocities to encourage physically plausible control.

  2. A grasp shaping term, rgrasp, encourages grasping and lifting the object (using terms like rapproach and rlift).

  3. The dominant reward is the goal-reaching term, rgoal, which drives progress toward a goal pose: "rgoal = max(d∗ − d(ot, g), 0) + Bsucc I[d(ot, g) < ϵ]." This sparse success bonus ensures the agent receives positive reinforcement only upon reaching a goal and then samples a new one.

Object-Centric Perception and Policy Inputs

To enable zero-shot transfer to novel tools, the policy is conditioned on object representations that are reliably available at deployment. The policy inputs include:

  1. The current 6D tool pose and a coarse 3D grasp bounding box, which encodes the intended graspable region.

  2. An LSTM backbone is used so the policy can integrate interaction history and implicitly infer latent physical and geometric properties that are not directly observed.

  3. The observation space includes proprioception, object state, goal information, and a coarse object descriptor ϕ (e.g., approximate geometry or physical properties).

Sim-to-Real Transfer Pipeline

The deployment pipeline bridges the sim-to-real gap using vision foundation models:

  1. From an RGB-D human video demonstration, SAM 3D is used to generate a metric-scale object mesh and segment a 3D grasp bounding box.

  2. FoundationPose is then used to extract a sequence of 6D goal poses. This trajectory is processed by temporal downsampling (from 30Hz to 3Hz) and lift-off truncation (starting at the first frame where the object's height exceeds a threshold, e.g., 10cm).

  3. During inference, the policy conditions on proprioception, current object pose, grasp bounding box, and goal pose. The goal advances only when the distance between the current pose and goal pose is sufficiently small, i.e., when "d(ot, g) < ϵ."

Evaluation and Generalization

Performance is evaluated using DexToolBench, a benchmark of daily tool-use behaviors consisting of 24 tasks across 6 categories (Hammer, Marker, Eraser, etc.) and 12 object instances. The evaluation measures Task Progress, defined as the percentage of demonstrated goal poses successfully tracked. SimToolReal demonstrates strong zero-shot generalization over 120 real-world rollouts spanning 24 tasks, 12 object instances, and 6 tool categories, outperforming prior methods using fixed grasps and motion retargeting by 37%. The study also shows a strong correlation between the training objective (random goal-pose reaching) and downstream performance on unseen tools.

RL Design Choices

Key RL design decisions were investigated to maximize performance:

  1. The choice of optimization algorithm, finding that SAPG mitigates exploration bottlenecks better than standard PPO by maintaining a population of policies.

  2. The use of an Asymmetric Critic, where the critic has access to privileged states in simulation (ground-truth state) while the actor operates on restricted observations, which is essential for overcoming partial observability.

Improvements for AI systems

Here are the specific improvements that can be made to AI systems by leveraging SimToolReal, along with a description of what these improved systems can achieve:

The core improvement lies in transitioning from task-specific, object-dependent RL policies to a single, generalist policy capable of zero-shot adaptation for novel tools and tasks. This capability is built upon three synergistic advancements: an object-centric training paradigm, a robust sim-to-real perception pipeline, and a universal goal-reaching objective.

Here are the specific improvements:

The resulting improved AI system can achieve the following:

Sources

Related papers