FlashDexRetarget: Accelerating Dexterous Manipulation Data Generation through Multi-Motion Retargeting

FlashDexRetarget introduces an RL-based framework for high-success, efficient dexterous motion retargeting by jointly training a single policy across multiple human hand-object demonstrations. This method addresses the limitations of existing physics-based and RL approaches—which typically require separate policies or independent optimization for each demonstration—by utilizing multi-reference tracking, geometry and interaction-aware observations, and dedicated per-hand actor-critic networks to achieve 100× lower training compute compared to evaluated baselines.

The gist

FlashDexRetarget is an RL framework that jointly retargets human hand–object demonstrations through multi-reference tracking with a single policy.

How it works

The core idea of FlashDexRetarget is to condition a single reference-conditioned policy on both the current simulated state and the next reference state from a collection of demonstrations, allowing for experience reuse across all references. The input to the policy at each step is defined by:

1. Current robot and object states: This includes current robot and object states denoted as x cur t.

2. Reference hand-object states: This includes the next target state and motion information over the next K steps, denoted as s ref i,t+1, which incorporates x ref i,t+1, Pref i,t+1, and z ref i,t+1.

To distinguish diverse interactions across references and provide temporal context beyond the immediate reference frame (where K=10), the framework employs several key observation components:

(a) Interaction-aware observations:

The framework uses object point clouds (Pcur t and Pref i,t+1) to represent object geometry in wrist-local coordinates. Furthermore, it computes distances from each fingertip and wrist to the object surface, concatenating these distances and corresponding surface normal vectors in hand’s wrist frame to form d cur t.

(b) Future reference conditioning:

To provide temporal context, the framework encodes the future reference sequence from t+1 to t+K using an encoder Encψ: z ref i,t+1 = Encψ(x ref i,t+1:t+K, mapping this sequence to a 128-dimensional latent representation.

Reward Structure

The policy is optimized using a composite reward function r h t for each hand h ∈ th hands. This reward aggregates four distinct supervision signals:

1. Object tracking reward (r obj,h t): This measures object misalignment by computing pointwise errors against the reference object pose, utilizing the mean of the three largest errors to emphasize the most significant object misalignments. The final reward is calculated as r obj t = exp(-e obj t / σobj).

2. Hand-object interaction reward (r int,h t): Instead of relying on discrete contact labels, this uses continuous hand–object distance signals instead of discrete contact labels. For each reference fingertip f, it computes the unsigned distance d ref,(f) t to the nearest object vertex and penalizes distances exceeding the reference distance: ε(f) t = max(0, d(f) t - d ref,(f) t). The reward is then defined as r int t = (1/5) Σ exp(-ε(f) t / σint).

3. Hand tracking reward (r track,h t): This combines exponential penalties on wrist position, wrist orientation, and mean fingertip position errors with a behavior-cloning term toward kinematic retargets.

4. Regularization reward (r reg,h t): This penalizes large action magnitudes and failures caused by the hand or object exceeding predefined distance thresholds.

Architecture and Learning Algorithm

FlashDexRetarget employs a specialized architecture to handle bimanual coordination efficiently:

1. Bimanual Architecture: Instead of a unified actor-critic pair, the framework maintains separate left- and right-hand actor critic networks (πθh and Qφh for h ∈ th hands). Each actor predicts the action for its respective hand, while both controllers observe the full bimanual state but control only their own hand.

2. Off-policy Learning: The framework adopts FlashSAC [11], an off-policy algorithm, which is scaled by increasing the replay buffer capacity from 10M to 50M transitions and the critic hidden dimension from 256 to 1024 to support the broader state distribution induced by multi-reference training.

Key Findings and Contributions

The evaluation on a benchmark of 50 motions spanning single-object and two-object interactions demonstrated significant performance gains:

(i) Efficiency:

FlashDexRetarget achieved a "

Here are the specific improvements that can be made to existing AI systems, based on the FlashDexRetarget paper, and what those improved systems could achieve:

1. Improvements in Data Efficiency for Dexterous Manipulation Transfer:

The core improvement is moving from single-reference optimization (which scales linearly with data size) to a multi-reference tracking framework using a shared policy.

2. Enhanced Retargeting Success Rate:

The system can achieve significantly higher success rates (up to 90% on the benchmark, compared to <10% for some baselines) when transferring complex human hand-object demonstrations across different robot embodiments (e.g., XHand and Sharpa Wave Hand).

3. Reduced Computational Cost for Data Generation:

The improved system requires up to 100x less training compute than evaluated RL-based baselines (like CHORD) to achieve comparable performance on the 50-motion benchmark, drastically lowering the cost of creating large datasets for robot learning.

4. Robustness to Diverse Interaction Geometries:

By incorporating object point cloud observations and hand-object distance features into the observation space, the system can better distinguish between different interaction geometries across various demonstrations, leading to more accurate trajectory generation in complex scenarios.

5. Improved Contact Modeling via Distance-Based Supervision:

The framework replaces unreliable exact contact matching with continuous, distance-based interaction rewards derived from object point clouds and Signed Distance Fields (SDFs). This allows the system to learn physically grounded motions even when human demonstrations contain imprecise or noisy contact information, improving robustness in simulation and real-world execution.

6. Better Temporal Context Utilization:

The inclusion of future reference conditioning (encoding states from t+1 to t+K) allows the policy to anticipate subsequent motion and maintain a more coherent interaction flow over time, leading to smoother and more temporally consistent robot trajectories.

7. Decoupled Bimanual Control for Asymmetric Tasks:

By employing separate actor-critic networks for the left and right hands, the system can effectively handle bimanual motions where the two hands have distinct roles or asymmetric learning signals (e.g., one hand gripping while the other stabilizes).

The improved AI system can be used to:

1. Generate large, high-quality synthetic datasets of robot manipulation trajectories from a limited collection of human demonstrations at a fraction of the computational cost.

2. Accelerate the training and fine-tuning of dexterity policies for robot hands by providing physically grounded, multi-reference data that is ready for deployment in simulation or real-world execution.

3. Enable robots to perform complex, coordinated tasks like tool use, container handling, and object assembly with high success rates across different robot hardware configurations.

Human hand-object demonstrations provide a scalable source of data for dexterous robot learning, but transferring them across embodiments requires physically feasible retargeting. Existing physics-based methods typically optimize each demonstration independently, leading to either limited success under finite simulation budgets or training costs that grow with dataset size. We introduce FlashDexRetarget, an RL framework for multi-reference dexterous retargeting. We formulate retargeting as multi-reference tracking, jointly learning a single policy across many demonstrations with off-policy RL and geometric supervision of the demonstrated interactions. This shared training formulation amortizes optimization across references while enabling the policy to track diverse hand-object interactions. On a 50-motion benchmark from TACO, OakInk2, and HOT3D using XHand and Sharpa Wave Hand as target embodiments, FlashDexRetarget retargets 90% of demonstrations using about 30 GPU-hours, compared with about 46% at about 3,000 GPU-hours for CHORD. This corresponds to about 100 times lower training compute and a 44-percentage-point improvement in retargeting success. Ablations examine the key design choices, while experiments with up to 1,000 motions and real-world replay further demonstrate the scalability and practical applicability of our method.