Generative Neural Retargeting for Human-to-Robot Dexterous Manipulation

arXiv:2610.12440 · cs.RO · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "Generative Neural Retargeting for Human-to-Robot Dexterous Manipulation".

Dev: The gist The proposed Generative Neural Retargeting (GNR) is a scalable and generalizable retargeting method that retargets large-scale, long-horizon and high-precision human demonstrations for five-fingered dexterous manipulation.

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: So we're looking at this paper now, "Generative Neural Retargeting for Human-to-Robot Dexterous Manipulation," and the authors are Gao, Yang, Abbatematteo, Godwin, Wang, Boldu, Manitoe. Basically they’re tackling that big problem where you have a human doing something in a video and you want to make a robot do it with high precision.

Dev: Yeah the core idea here is that existing methods for making robots follow human motions often get stuck because they ignore the physics or dynamics, leading to trajectories that just don't work in reality. They’re proposing this Generative Neural Retargeting method as a way around those issues.

Taro: What I find interesting about this setup is how they are trying to bridge that gap between a video of a person moving and the actual physical movement required by the robot, which is usually where things get messy.

Rosa: Exactly. This paper talks about using an AI model to generate those feasible motions directly from human motion data, rather than just guessing based on simple geometry or basic inverse kinematics. They use something called flow matching to sample trajectories that are actually possible in the real world.

Dev: And they frame it by saying that dynamically feasible trajectories tend to cluster around a low-dimensional area, which means they aren't scattered everywhere in possibility space; the AI learns where those feasible motions live.

Taro: So instead of brute-forcing a solution for every single trajectory, the system learns from a tiny starting set of data generated by another method to guide the sampling process toward things that actually work.

Rosa: Right. The paper describes how they model the difference between the desired motion and what's already feasible, which they call residual corrections, using this conditional flow matching generator Gtheta.

Dev: It sounds like they are training this generator to predict those correction terms based on the human motion and some info about the object or task, which is what’s encoded in that c parameter.

Title and authors: Taro: I wonder how robust this gets when the robot encounters something completely different from what it was trained on, because that’s always a big worry with these kinds of generative models.

Rosa: That’s where they show their scaling capabilities. They built this whole system using a real-to-sim data engine that can take videos and turn them into millions of demonstrations and object geometries, including contact forces for the hands.

Dev: The data engine part is huge; it generates a dataset with 223k demonstrations and 3 point 3k object geometries, which is necessary to give this generative model enough variety to learn from.

Taro: And they don't just collect data; they have a pipeline for expanding that data by changing the object physics and geometry, even regenerating meshes from the same image with different random seeds.

Rosa: That expansion process helps them cover more ground on both the diversity axis—how many different things it can do—and the difficulty axis, which is about how precise or long-horizon the task is.

Dev: On that diversity axis, they show that Generative Neural Retargeting outperforms sampling methods like MPC by using only about eight point five percent of their sample budget on HOTthree dee after learning from a small seed dataset generated by MPC <ref:2610.12440#pg3,budget on HOT3D after learning from a small seed dataset generated by>.

Taro: That’s a big number because it means the AI is much smarter about finding good paths than just letting the sampler run wild with more samples, which is what MPC usually does.

Rosa: And when you look at out-of-distribution objects, motions, and geometries—things the model hasn't seen—GNR achieves a fifty-six point two percent success rate compared to around thirty percent for those sampling-based methods <ref:2610.12440#pg3>.

Dev: That jump in success on unseen things really shows its generalization power beyond just the data it was explicitly trained on, which is a key thing for real-world deployment.

Taro: For the difficulty axis, they also solve forty-two point six percent of demonstrations that MPC fails on when using the sample budget that generated this GNR training data <ref:2610.12440#pg3>. That points to its ability to handle those hard, specific tasks better than the baseline controllers used with MPC.

Title and authors: Rosa: And for precision tasks, like long-horizon or millimeter-precision ones in things like the NIST task, they can solve them open-loop just by giving MPC an initialization that's already on a good manifold.

Dev: That initialization is generated using this learned residual correction, equals G theta(z zero c), which means you’re adding a learned correction term to the standard control input, u zero:T - one = zero:T - one +.

Taro: So it’s not just about finding a path; it’s about learning the specific, dynamic adjustments needed to make that path actually execute correctly when you have physical constraints.

Rosa: The efficiency gains are also quite striking. They report that GNR reduces the CPU cost per trajectory by seven point six times compared to MPC on things like HOTthree dee, which is a significant reduction in computational load for real-time systems <ref:2610.12440#pg3,CPU cost per trajectory by 7.6>.

Dev: That efficiency comes from how they use the learned model at inference time, where you just draw some noise and let the generator produce the correction, instead of running a whole optimization loop every single time.

Taro: It seems like this is moving manipulation away from needing massive amounts of dedicated training data or incredibly slow optimization steps toward having a system that can quickly generate feasible actions on demand.

Rosa: Right. But they also have to be careful with what they claim about the data engine pipeline, because scaling up that initial dataset creation is still a big part of the picture.

Dev: And as we move into the next part of this paper, we’ll look at how they address those issues and what these results mean for actual robot deployment outside of a perfect simulation setting.

Taro: I'm curious to hear more about those practical limitations, because in robotics, a model that works perfectly in simulation can often fall apart when it hits real-world sensor noise or unexpected physical interactions.

Rosa: That’s what we’ll get to next as we look at the improvements they suggest and how this all fits into the bigger picture of robot learning.

The paper's summary: Rosa: So, basically what we're looking at here is this Generative Neural Retargeting method that takes human demonstrations and turns them into precise robot motions, not just some rough guess at how a hand should move.

Dev: It learns to generate those specific corrections needed to bridge the gap between the human doing the task and what the robot actually has to do physically.

Taro: The big takeaway is that this isn't just another sampling technique; it’s about using generative models, specifically flow matching, to sample trajectories that are dynamically feasible right from a small starting point generated by something like MPC.

Rosa: It means instead of running a slow optimization process for every single trajectory we want the robot to take, the AI learns where those successful motions live in some low-dimensional space and just samples from there.

Dev: And they frame it this way, modeling those differences between the desired motion and what's already feasible as residual corrections, which is conditioned on things like where the object is or how it looks.

Taro: What I find interesting for autonomy research is how they build that data engine to scale up things. They don't just rely on one video; they create a massive dataset with hundreds of thousands of demonstrations and thousands of different object shapes, including contact forces.

Rosa: That data engine pipeline is pretty smart because it has these discovery and scaling phases, where it filters for successful movements under various conditions, then expands them by changing the object physics or even regenerating the three dee meshes.

Dev: That expansion is what lets them test how much of this system works when you introduce completely new objects or motions that weren't in the original training set.

Taro: They show they handle out-of-distribution things much better than traditional sampling methods, achieving higher success rates even when the robot sees something it hasn't encountered before.

Rosa: It’s also about task difficulty; they say this method can solve long-horizon or millimeter-precision tasks that other controllers just can't handle with their current sample budgets.

Dev: And for those hard, precise tasks like the NIST challenge, they can supply an initial control sequence that’s already on a good path, which then lets the robot fine-tune it to get that high success rate.

Taro: So what this means for us in autonomy is that we might not need endless hours of manual tuning or massive datasets just to get a robot to perform a complex manipulation task correctly.

Rosa: It shifts the focus from purely data collection and brute force optimization toward learning the underlying structure of feasible motion directly from expert demonstrations.

Dev: And computationally, they actually show some efficiency gains; for instance, reducing the CPU time needed per trajectory compared to running a standard MPC loop by about seven times on certain benchmarks.

Taro: That speed is crucial because you need that kind of fast correction when the robot is interacting with something in real-time.

Rosa: But they also have to be careful about what this learns; it’s learning from human demonstrations, so how well it transfers to a completely different physical setup outside of those training conditions is still an open question.

Dev: Exactly. The next thing we should look at is how they handle that transition from simulation training to the messy reality of a physical robot.

The paper's improvements: Rosa: So we’re talking about what they suggest to make this Generative Neural Retargeting method even better than it is right now.

Dev: They are focused on improving how test-time scaling actually works, because that’s where I worry about loop rates and latency issues in a real robot.

Taro: The authors point out that test-time scaling isn't just about increasing the number of samples; it’s more about using those extra samples to improve the success rate on things the model hasn't seen, like really weird objects or movements.

Rosa: Right, so they suggest that instead of just throwing in more random noise for sampling, you use that scaling budget to actually guide the sampling process toward better parts of the feasible motion manifold.

Dev: That makes sense because if you just sample more randomly, you’re still wasting computation time on trajectories that are guaranteed to fail anyway.

Taro: They also touch on using MPC initialization, which is this clever idea where they supply a high-quality starting point based on what the GNR model has learned about feasible motions.

Rosa: So instead of starting from scratch with a generic guess, you start with something that’s already dynamically sound according to the AI’s understanding of human motion.

Dev: That initialization helps stabilize the whole process, which is important for keeping our control loops smooth and avoiding sudden failures when things get tricky in the field.

Taro: They also hint at future work regarding sim-to-real transfer, because getting this learned policy to work reliably when you move it out of the controlled simulation environment is always a huge hurdle.

Rosa: That’s the big question for field robotics; if we can train something that works perfectly in a simulator but fails when it hits real-world sensor noise or unexpected physical interactions, then the whole thing doesn't solve much.

Dev: So they are pushing toward systems where the learning process itself is more robust to those kinds of discrepancies, which brings us to how this fits into broader learning frameworks.

Taro: It makes me think about how this relates to things like STEAM or RA-VLA, where the goal is often to make sure the policy can adapt its behavior on the fly when it encounters something unexpected during operation.

Rosa: Exactly. This GNR framework seems designed to give us a strong foundation for that adaptation, but we need to see how it integrates with those other learning frameworks we’ve been discussing.

Dev: I want to see if this learned correction mechanism can be easily plugged into existing control architectures without messing up the established loop rates or adding too much computational overhead.

Conclusion: Rosa: So to wrap up this session, we've been looking at how Generative Neural Retargeting for Human-to-Robot Dexterous Manipulation uses flow matching to generate feasible motions from human demonstrations.

Dev: It’s a method that scales better than traditional optimization by learning the structure of physically possible trajectories through residual corrections conditioned on the task and object.

Taro: The main implication for autonomy is that we can get much higher success rates on difficult, out-of-distribution tasks because the AI learns where those feasible motions live in a low-dimensional space.

Rosa: And for anyone listening who just wants to know what this means, it’s that robots can perform complex manipulation tasks with less training data and less brute force searching than before.

Dev: The efficiency gains are real too; they show a significant reduction in the computational cost per trajectory when you compare it against older sampling-based methods like MPC.

Taro: But we have to remember the caveats, Rosa, because this is still trained on human data, and that raises questions about how well it transfers when deployed in a completely new physical environment.

Rosa: True. The authors themselves flag that while the scaling works well within their specific real-to-sim engine pipeline—handling two hundred twenty-three thousand demonstrations and thousands of geometries—the sim-to-real gap is still a major challenge.

Dev: That’s why I think the next thing we need to look at is how they address those gaps directly, specifically how to make that learned correction robust against real-world sensor noise and unexpected physical interactions.

Taro: And that leads perfectly into what I wanted to talk about next, because we have this other paper called RA-VLA which deals with test-time adaptation, and it seems like a natural follow-up discussion for how this kind of generative modeling interacts with on-the-fly policy refinement.

Rosa: Exactly. So while the Generative Neural Retargeting for Human-to-Robot Dexterous Manipulation shows great promise in generating high precision, we still need to figure out how to make that generation robust enough to handle the messy reality of real robot deployment.

Dev: It's a big step toward making AI agents more capable of complex physical tasks, but the engineering challenge is definitely moving it from a clean simulation environment into the unpredictable world.

Dechen Gao, Yue Yang, Ben Abbatematteo, Nathan Godwin, Pengcheng Wang, Roger Boldu, Steven Man, Zhiyang Dou

Meta Reality Labs Research 2UC Davis University of California Davis, UNC Chapel Hill, UC Berkeley, MIT, CMU

cs.RO

Submitted: 2026-10-08

Updated: 2026-10-08

The gist: The gist The proposed Generative Neural Retargeting (GNR) is a scalable and generalizable retargeting method that retargets large-scale, long-horizon and high-precision human demonstrations for

Key concepts

Flow Matching Model
This is the core generative model used by GNR. It learns a continuous path between a simple noise distribution and the desired trajectory, conditioned on human motion. By modeling residual corrections as a conditional flow, it efficiently samples dynamically feasible movements without needing extensive optimization for every task.
Residual Correction ($Δ$)
This represents the difference between an initial control sequence and the target reference control sequence. GNR learns to predict these small corrections based on human motion and object properties. This allows the model to refine a base trajectory into a precise, feasible path that adheres closely to human demonstrations.
Data Expansion Pipeline
This two-phase process scales the training data for egocentric demonstrations. The discovery phase filters successful segments based on performance metrics, while the scaling phase expands these successful segments by varying object physics and geometry, ensuring the model learns robust retargeting across diverse real-world scenarios.

Terminology

Summary

The gist The proposed Generative Neural Retargeting (GNR) is a scalable and generalizable retargeting method that retargets large-scale, long-horizon and high-precision human demonstrations for five-fingered dexterous manipulation.

How it works

Generative Neural Retargeting (GNR) uses a flow matching model to sample feasible trajectories conditioned on human motion. This approach addresses the scaling bottleneck of conventional sampling-based optimization by hypothesizing that dynamically feasible trajectories concentrate near a low-dimensional manifold shared across demonstrations. GNR learns from a small seed dataset generated by MPC to guide sampling toward dynamically feasible trajectories, conditioned on human motion.

The core mechanism of GNR involves modeling the residual corrections as a conditional flow matching generator Gθ(· c) of residual corrections ∆ = u0:T −1 − u¯0:T −1 to the IK reference controls u¯0:T −1, conditioned on c = (x¯0:T, o), where o encodes object geometry or pose depending on the task. The training involves sampling standard Gaussian noise z0 ∼ N (0, I) and flow time τ ∼ U[0, 1], setting zτ = (1−τ)z0 +τ∆, and training the velocity field vθ with the flow matching loss LFM(θ) = E(∆,c),z0,τ∥vθ(zτ, τ, c) − (∆ − z0)∥2. At inference, Gaussian noise z0 is drawn and integrated to yield the residual ∆ˆ = Gθ(z0 c) = zˆ1 and the retargeted trajectory u0:T −1 = u¯0:T −1 + ∆ˆ.

Data Engine and Scaling

GNR is applied within a real-to-sim data engine that reconstructs human motion from egocentric videos or a wearable exoskeleton with motion capture to scale up both diverse and difficult high-precision tasks. This engine produces a dataset spanning 223k demonstrations and 3.3k object geometries, including dense contact-force labels. The data engine takes per-frame human hand motion, object poses, and an object surface mesh M as input.

The data expansion pipeline involves a two-phase process for egocentric demonstrations. The discovery phase identifies retargetable segments by optimizing under nominal conditions and sixteen single-axis perturbations, retaining only segments with a success criterion Γ(ξ) = γtrack(ξ) ∧ γpen(ξ) ∧ γgrasp(ξ) ∧ γforce(ξ). The scaling phase expands these successful segments by varying object physics and geometry, including mesh regeneration from the same image with different random seeds, anisotropic deformation, and single-axis stretching.

Performance Across Axes

GNR demonstrates superior performance along both the diversity and difficulty axes compared to sampling-based methods like MPC. Along the diversity axis, GNR outperforms MPC with about 8.5% of their sampling budget on HOT3D after learning from a small seed dataset generated by MPC. On out-of-distribution (OOD) objects, motions, and geometries, GNR reaches 56.2% success versus around 30% for sampling-based methods.

Along the difficulty axis, GNR solves long-horizon and millimeter-precision tasks open-loop or by supplying MPC with on-manifold initializations. For instance, GNR solves 42.6% of demonstrations on which MPC fails under the sampling budget used to generate GNR’s training data. Test-time scaling complements data scaling by improving success through sampling, reducing reliance on costly seed datasets.

Efficiency and Deployment

GNR significantly reduces CPU cost per trajectory compared to MPC. For HOT3D, GNR reduces CPU cost per trajectory by 7.6× relative to MPC. In the NIST task, GNR solves long-horizon, millimeter-precision tasks open-loop or by supplying MPC with on-manifold initializations.

The efficiency analysis shows that using a smaller seed dataset can reduce the MPC generation cost significantly. For instance, for HOT3D, using 28k MPC seeds requires about 116.1k core-h for seed generation and 32.6k core-h for GNR generation, resulting in a total cost of 148.7k core-h. This represents a 2.62× reduction in cost relative to DIAL-MPC.

Conclusion

GNR generalizes across motions and geometries, achieves higher success than MPC with far fewer samples, and retargets demonstrations on which MPC fails. GNR enables test-time scaling to complement data scaling when feasible training trajectories are costly. Future work may study sim-to-real transfer of the learned policy.

Improvements for AI systems

  1. Bold header: Generative Neural Retargeting (GNR) for scalable trajectory sampling. GNR learn[s] from a small seed dataset generated by MPC to guide sampling toward dynamically feasible trajectories, conditioned on human motion, which reduces the reliance on expensive per-trajectory optimization and allows for efficient retrieval of feasible motions.

  2. Bold header: Improved sample efficiency across diversity and difficulty axes. GNR outperforms MPC with only 8.5% of the samples required by MPC (on HOT3D), and it solves demonstrations on which MPC fails under the sampling budget used to generate GNR’s training data.

  3. Bold header: Enhanced generalization to unseen scenarios via test-time scaling. The system demonstrates that test-time scaling complements data scaling: increasing the sampling budget improves retargeting success while reducing reliance on costly seed data, allowing GNR to succeed where MPC fails on OOD objects, motions, and geometries.

  4. Bold header: Scalable real-to-sim data engine for large-scale dataset creation. The proposed engine can produce a dexterous manipulation dataset with dense contact-force labels, spanning 223k demonstrations and 3.3k object geometries by applying GNR within the pipeline to scale both diversity and difficulty.

  5. Bold header: High-precision task capability via on-manifold initialization. For long-horizon tasks like NIST, GNR can supply a high-quality on-manifold initialization, u0:T −1 = u¯0:T −1 + ∆ˆ, which MPC then refines to reach higher success rates than vanilla MPC.

  6. Bold header: Efficient data generation cost reduction via seed set utilization. By using a smaller seed dataset, the system achieves significant cost savings; for instance, for NIST, using 100 seed demonstrations reduces generation cost from 79.14k to 44.52k core-h at about 92.2% success.

  7. Bold header: Policy learning applicability via behavior cloning. GNR-generated data is applicable to policy learning, as the best SR achieved by the policy trained using MPC data is 37.5%, 43.75% using GNR+MPC, and 43.75% using GNR.

Sources

Related papers