UniCross: Unified Cross-Skill Dexterous Manipulation Synthesis

arXiv:2607.28198 · cs.RO, cs.CV · Submitted 2026-07-30 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "UniCross: Unified Cross-Skill Dexterous Manipulation Synthesis".

Dev: A unified framework for cross-skill dexterous manipulation synthesis has been presented that models grasping, relocation, in-hand rotation, and in-hand translation from a shared hand-object relational perspective.

Rosa: First, who's behind it and why it matters.

Paper summary: Rosa: So we're looking at the paper "UniCross: Unified Cross-Skill Dexterous Manipulation Synthesis," and it seems like the authors are proposing a way to model grasping, relocation, in-hand rotation, and in-hand translation all from a single hand-object relational perspective. This unified formulation is what allows them to distill one policy that can handle every skill well and generalize to new objects. It really claims this approach solves the problem where existing methods use skill-specific designs that break continuity when you try to chain those skills together <ref:2607.28198#pg0>.

Dev: I see the core thesis is about moving away from designing separate policies for each skill, which they say is what breaks the compatibility and continuity for skill composition <ref:2607.28198#pg1>. They are trying to find a way to capture the shared structures across these different manipulation behaviors instead of relying on isolated execution <ref:2607.28198#pg0>.

Taro: From an autonomy standpoint, what I find interesting is how they're tackling the difficulty of unifying state spaces and reward structures when these skills demand fundamentally different contact regimes, like grasping needing stable contacts versus rotation needing frequent reconfiguration <ref:2607.28198#pg1>. If you can unify those things, it opens up a lot more possibilities for the AI to handle complex, unforeseen situations.

Rosa: Exactly, and what matters is that they formulate these different behaviors as "different instantiations of a single formulation conditioned on desired relational motions" <ref:2607.28198#pg0>. That suggests the underlying mathematical structure is the same, but the conditions change depending on which skill you want to perform. This should lead to much better generalization than what we see now <ref:2607.28198#pg2>.

Dev: And that unified formulation requires a unified observation space, denoted as o t = (s h t, s o t, g t), which includes the hand state, an interaction-aware object representation like

c t, f t, v t: , and relation-based objective features <ref:2607.28198#pg0>. That's a lot of information to manage for the loop rate.

Taro: The description of s o t as capturing "per-link contact states c t, contact force magnitudes f t, and the distance vectors v t from each finger link to their nearest points on the object surface" seems quite detailed for a unified representation <ref:2607.28198#pg1>. How does that specific way of encoding contact information help bridge those skill differences we talked about?

Rosa: It seems that by including these physical, contact-based features directly into the state, they are building the necessary foundation for a shared understanding across all four skills <ref:2607.28198#pg0>. This is crucial because they acknowledge that grasping and relocation require persistent stable contacts while rotation and translation need different kinds of dynamic contact adjustments <ref:2607.28198#pg1>.

Dev: The action space is also unified, parameterizing the full hand motion by outputting incremental motion commands that get transformed into target joint positions q act t = clamp(q ref t + alpha times a t, q, q) <ref:2607.28198#pg0>. That suggests they're aiming for a continuous control signal that can drive any of these motions. I wonder about the latency implications when you have this complex state input and output command structure running in real-time <ref:2607.28198#pg0>.

Paper summary: Taro: If the action space is unified, it implies that the policy doesn't need to learn entirely new control strategies for every skill; it just needs to condition that single strategy differently based on the relational motion goals <ref:2607.28198#pg0>. That level of abstraction could allow for much faster adaptation when the environment throws unexpected physics at the system.

Rosa: And they support this by using a shared reward structure r t = r goal t + r track t + r reg t, where the goal term encourages stable contact and motion along a target axis, and the regularization term penalizes pose deviations, wrist velocity, applied torque magnitude, and drop penalties <ref:2607.28198#pg0>. It seems they've tried to bake stability into the objective function itself.

Dev: That reward structure is pretty comprehensive; those penalties for pose deviations and applied torque magnitude are interesting because they directly address the physical stability concerns we have in hardware deployment <ref:2607.28198#pg0>. I'm curious how they tune those penalty weights to balance task achievement against maintaining a low-energy, stable state during long sequences.

Taro: Considering the context of long-horizon manipulation, this shared objective structure must be what enables the skill chaining they aim for <ref:2607.28198#pg0>. If the reward function is consistent across grasping and then moving that grasped object, it should naturally guide the policy to connect those actions without needing explicit transitions between skill models <ref:2607.28198#pg1>.

Rosa: And they address the fact that unifying different skills imposes higher requirements on object shape representation, noting that grasping needs to identify regions for stable contact forces while in-hand manipulation requires dynamic shape features <ref:2607.28198#pg1>. It sounds like the unified representation s o t is specifically designed to capture those diverse geometric necessities <ref:2607.28198#pg0>.

Dev: The distillation process they use, where ten per-skill policies are trained separately and then distilled via vanilla DAgger using MSE imitation loss, sounds like a clever way to leverage the expertise from individual skill models before merging them <ref:2607.28198#pg2>. But how does that MSE loss handle the differences in the underlying dynamics between those ten distinct policies?

Taro: It seems they are relying on the pre-trained experts to provide good starting points, and then using distillation to ensure continuity when combining them into one policy <ref:2607.28198#pg0>. This suggests that having strong skill-specific baselines is still valuable, even if the final goal is a single unified controller <ref:2607.28198#pg2>.

Rosa: The performance metrics they show suggest this works very well, particularly concerning generalization to unseen geometries and maintaining robustness under external perturbations <ref:2607.28198#pg3>. That's the kind of practical validation we need to see before we think about deploying this in a real-world setting outside the lab <ref:2607.28198#pg3>.

Paper summary: Dev: Robustness to disturbances is critical; if the hand can stably hold an object even under persistent perturbations, that significantly lowers the failure modes for any physical robotic system <ref:2607.28198#pg3>. That stability must be rooted in how they handle those contact forces and motion constraints we discussed earlier.

Taro: And regarding the long-horizon aspect, the framework's ability to support skill chaining through state compatibility is what really makes this relevant for complex tasks <ref:2607.28198#pg0>. If a system can seamlessly execute grasp, relocate, and then rotate without needing a separate planning module for each step, that simplifies the overall autonomy stack significantly <ref:2607.28198#pg1>.

Rosa: So to wrap up this part of the presentation, UniCross is presented as a coherent framework that uses a shared hand-object relational perspective to unify four key manipulation skills under one policy <ref:2607.28198#pg0>. It moves beyond skill-specific designs by formulating behaviors based on desired motions <ref:2607.28198#pg1>.

Dev: And the distillation technique they employ, training a single policy from ten experts via MSE loss, is the mechanism they use to achieve that cross-skill compatibility and continuity for long sequences <ref:2607.28198#pg2>. It seems like a solid way to handle the complexity of unifying those different skill dynamics.

Taro: The implications here are that we might be able to build generalist manipulators that can perform a wide variety of complex tasks simply by training on this unified framework, rather than needing task-specific training for every single application <ref:2607.28198#pg2>. This points toward a more adaptable autonomous agent.

Rosa: It really suggests that the next step is seeing how robust this performs when taken outside the simulation, and for how long it can actually maintain that performance in physical environments <ref:2607.28198#pg3>. That's where we need to push the boundaries of this work.

Dev: And from an engineering standpoint, we need to confirm that the required loop rate for processing that unified observation space and generating those action commands is feasible on current hardware <ref:2607.28198#pg0>. Latency management will be key to realizing this in a real system.

Taro: If the system encounters something completely novel, like an object with an unexpected geometry that doesn't fit the learned relational patterns, we have to consider how its current formulation handles that misbehavior <ref:2607.28198#pg1>. That's a key area for future work regarding true world interaction.

Rosa: So, looking at the overall structure of UniCross: Unified Cross-Skill Dexterous Manipulation Synthesis, it’s about creating a single policy that generalizes across skills and handles long sequences by sharing a common relational model <ref:2607.28198#pg0>. It's ambitious in its goal to create one system that doesn't require separate designs for every manipulation action we can think of.

Paper summary: Dev: The authors are focusing heavily on the shared state and action spaces, which is what underpins the ability to chain those skills together successfully <ref:2607.28198#pg0>. That structural consistency is what makes the skill composition feasible in their model.

Taro: If this framework proves reliable in complex, long-horizon tasks, it has implications for deploying autonomous robots in unstructured environments where they need to perform sequences of actions that are not pre-programmed <ref:2607.28198#pg1>. It moves the focus from teaching specific movements to teaching relational goals.

Rosa: We really need to see if this can operate reliably outside the controlled simulation environment and maintain performance over extended periods in physical settings <ref:2607.28198#pg3>. That practical validation is where we'll be focusing our efforts next <ref:2607.28198#pg3>.

Dev: I agree that the engineering reality of real-time performance and failure mode handling is going to be a huge part of the next phase of research, especially with that unified state representation <ref:2607.28198#pg0>. We'll need to rigorously test those stability metrics we discussed earlier.

Taro: And if the AI can handle disturbances robustly, as they claim under external perturbations, it could mean robots don't have to be perfectly calibrated every single time they interact with the world <ref:2607.28198#pg3>. That level of inherent resilience is what makes true autonomy possible.

Rosa: So that's the core concept of UniCross: unifying grasping, relocation, rotation, and translation under a shared relational formulation to enable a single policy for long-horizon manipulation <ref:2607.28198#pg0>. It’s about making complex hand movements consistent across different skills.

Dev: And the distillation from ten per-skill policies is the technique used to synthesize that unified policy, which relies on training on aggregated state-action pairs using an MSE loss <ref:2607.28198#pg2>. That's a key part of how they bridge those skill gaps.

Taro: The big picture here is that this approach shifts the paradigm from learning isolated skills to learning a unified way to achieve relational goals, which opens up new avenues for generalist manipulation systems <ref:2607.28198#pg2>. It’s about building agents that understand how objects move relative to the hand, not just how to perform isolated actions.

Rosa: So, in summary, UniCross is a framework proposing a unified way to model and synthesize cross-skill dexterous manipulation through a shared hand-object relational perspective <ref:2607.28198#pg0>. It aims for one policy that handles every skill well and generalizes by using distillation from specialized experts <ref:2607.28198#pg2>.

Dev: And the success in generalizing to unseen geometries and handling disturbances suggests a high degree of robustness, which is vital for moving this beyond controlled lab settings <ref:2607.28198#pg3>. We'll be looking closely at the specifics of that robustness testing.

Taro: The implications are that we could see robots performing complex sequences much more reliably in real-world scenarios where they have to adapt their interaction based on the object they are holding <ref:2607.28198#pg3>. It suggests a path toward more capable, generalist robotic agents.

Conclusion: Rosa: So, we've been digging into UniCross to see how they unify grasping, relocation, rotation, and translation under one policy framework. Dev, what are your thoughts on the title and who came up with this work?

Dev: The title itself is pretty descriptive; it tells you exactly what the paper is about—a unified approach to cross-skill manipulation. As for the authors, I see a team that’s clearly deep into robotics and control systems, which suggests they have a solid foundation for tackling these complex motion problems.

Taro: I think the authors are really aiming at solving that compatibility issue we talked about earlier; they're trying to build something coherent instead of just stitching together separate skill solutions. It points toward a more holistic way of thinking about robotic tasks.

Rosa: That holistic view is what really gets me excited; it means we might see robots capable of doing much longer, more complex sequences without needing separate programming for each step. What are the actual implications if this works well outside the lab?

Dev: The implication is that if the state representation and action space are truly shared, you could potentially deploy a single controller on hardware that needs to handle varied object interactions in real-time. I’m still focused on how fast that whole system can run without introducing unacceptable latency during those skill transitions.

Taro: If it can handle disturbances robustly, as the paper suggests, then these robots won't be so fragile when they encounter unexpected physics in a real environment. That level of resilience is what moves us closer to truly autonomous systems operating in messy settings.

Rosa: It sounds like the big picture here is moving from teaching specific movements to teaching a general method for achieving relational goals. Where should we look next to see if this kind of unified system can handle those long-horizon tasks?

ETH Zurich, Switzerland · inspire AG, Zurich, Switzerland

cs.RO, cs.CV

Submitted: 2026-07-30

Updated: 2026-10-07

Comments: Project page: https://zdchan.github.io/UniCross/

Project page: https://zdchan.github.io/UniCross

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 83/100

The gist: A unified framework for cross-skill dexterous manipulation synthesis has been presented that models grasping, relocation, in-hand rotation, and in-hand translation from a shared hand-object

Key concepts

Unified Observation Space: This is the combined input the system receives, consisting of three parts: the current state of the hand (s_h^t), an interaction-aware representation of the object (s_o^t) describing contact forces and distances, and relation-based features (gt). This comprehensive view allows the policy to understand both what it is holding and how it relates to that object.
Shared Reward Structure: The reward function is designed to guide all four skills simultaneously. It combines a goal term encouraging stable contact and desired object motion with a regularization term that penalizes instability, such as pose deviations, high wrist velocity, and excessive applied torque. This ensures the learned behavior is not only successful but also physically robust.
Skill Chaining: The framework naturally supports long manipulation sequences by ensuring that the output state of one skill is compatible with the input state of the next. Because all skills share a common formulation, a policy can transition smoothly from grasping to relocating or rotating without needing separate training for each step, enabling complex, multi-step tasks.

Terminology

Generalization — 18× in this paper · explained in 13 episodes
Action Space — 17× in this paper · explained in 1 episode
Distillation — 17× in this paper · explained in 2 episodes
Robustness — 12× in this paper · explained in 5 episodes
Computer Vision and Pattern Recognition — 10× in this paper
Unified Framework — 10× in this paper · explained in 5 episodes
Consistency — 6× in this paper · explained in 3 episodes
ACM Transactions on Graphics — 4× in this paper
Regularization — 4× in this paper · explained in 3 episodes
Ablation — 3× in this paper · explained in 1 episode
Core — 3× in this paper · explained in 2 episodes
Embeddings — 2× in this paper · explained in 2 episodes
In Conference on Robot Learning — 2× in this paper
Reinforcement Learning — 2× in this paper · explained in 4 episodes
Transferability — 2× in this paper · explained in 4 episodes
Ablation study — 1× in this paper · explained in 1 episode
Abstraction — 1× in this paper · explained in 1 episode
Adaptation — 1× in this paper · explained in 1 episode
Benchmark — 1× in this paper · explained in 2 episodes
Benchmarking — 1× in this paper · explained in 3 episodes
Diffusion Model — 1× in this paper · explained in 8 episodes
Diffusion Models — 1× in this paper · explained in 6 episodes
Downstream Tasks — 1× in this paper · explained in 2 episodes
Failure Modes — 1× in this paper · explained in 3 episodes
Generative Model — 1× in this paper · explained in 2 episodes
Generative Models — 1× in this paper · explained in 2 episodes
Grounding — 1× in this paper · explained in 3 episodes
Motion Primitives — 1× in this paper · explained in 2 episodes
Objective Function — 1× in this paper · explained in 2 episodes
Physical Constraints — 1× in this paper · explained in 7 episodes
Pruning — 1× in this paper · explained in 4 episodes
Reinforcement Learning (RL) — 1× in this paper · explained in 25 episodes
Reward Function — 1× in this paper · explained in 2 episodes
Reward Shaping — 1× in this paper · explained in 2 episodes
Tokenization — 1× in this paper · explained in 2 episodes
alpha — 1× in this paper · explained in 1 episode

Summary

A unified framework for cross-skill dexterous manipulation synthesis has been presented that models grasping, relocation, in-hand rotation, and in-hand translation from a shared hand-object relational perspective. This coherent formulation enables the distillation of a single cross-skill policy that performs strongly on every skill, generalizes to unseen objects, stays robust to disturbances, and chains skills seamlessly into long-horizon manipulation.

How it works

The framework models the four canonical skills—grasping, relocation, in-hand rotation, and in-hand translation—under a shared hand-object relational formulation that shares the same state and action spaces and a common objective structure. This approach moves away from skill-specific designs by formulating different behaviors as different instantiations of a single formulation conditioned on desired relational motions.

The core components of the unified framework include:

A unified observation space, denoted as ot = (s h t, s o t, gt), which includes the hand state (s h t), interaction-aware object representation (s o t), and relation-based objective features (gt). The object is represented with s o t = [c t, f t, v t], capturing per-link contact states ct, contact force magnitudes ft, and the distance vectors vt from each finger link to their nearest points on the object surface.
A unified action space that parameterizes the full hand motion by outputting incremental motion commands at that are transformed to target joint positions q act t = clamp(q ref t + α · a t, qmin, qmax).
A shared reward structure defined as rt = r goal t + r track t + r reg t, where the goal term (r goal t) encourages stable hand-object contact and object motion along the target axis, and the regularization term (r reg t) promotes stability through penalties like pose deviations, wrist velocity, applied torque magnitude, and a drop penalty.

Task Formulation

The four skills are characterized from the perspective of hand-object relational movements in two frames:

(Grasp):

The object is fixed in the root frame, while the hand moves in the root frame towards the object. The target poses are set such that the object is fixed in the root frame, while the hand moves in the root frame towards the object.

(Relocate):

The object is fixed in its current hand-frame pose, and the wrist moves in the root frame to reach a target object pose.

(Rotate):

The wrist is fixed in the root frame, and the object orientation rotates about a designated axis in the hand frame.

(Translate):

The wrist is fixed in the root frame, and the object position translates along a designated axis in the hand frame.

Unified Policy Distillation

To achieve cross-skill compatibility and continuity for long-horizon manipulation, ten per-skill policies are first trained independently using PPO [46]. These policies share the same observation space, network architecture, and action space. Subsequently, a single unified policy is distilled from these experts via vanilla DAgger [45]. This distillation process involves rolling out the distilled policy to collect states, query the corresponding per-skill expert actions for the visited states, and train the unified policy on the aggregated state-action pairs using an MSE imitation loss.

Performance and Generalization

Experiments demonstrate strong performance across all skills. The unified cross-skill policy consistently outperforms per-skill baselines under both original and general settings. Key findings include:

Strong generalization to unseen geometries, where the policy exhibits shape- and scale-adaptive manipulation strategies, such as adapting finger motion patterns based on object geometry (e.g., smooth wrapping for round objects vs. edge-aware periodic motions for prisms).
Morphology-agnostic transferability, achieving overall good performance among all three hands (MANO, Allegro, and Sharpa Wave) when trained with a unified policy.
Robustness to disturbances; the policy remains strong in this more difficult general setting under external perturbations, indicating the hand can stably hold the object even under persistent perturbations.

Long-Horizon Manipulation

The framework naturally supports long-horizon manipulation by enabling skill chaining due to state compatibility. The policy is evaluated on sequences such as grasp + relocate + rotate (Gr. + Re. + Rot.) or grasp + relocate + translate (Gr. + Re. + Trans.). The results show high success rates for these sequences, demonstrating the "cross-skill capability and state compatibility enabled by our unified framework.

Improvements for AI systems

As a fastidious and diligent researcher, I have analyzed the UniCross framework presented in this paper. The core innovation lies in moving from skill-specific constraints to a unified, hand-object relational formulation for dexterous manipulation.

Here are the specific improvements that can be made to AI systems by implementing or leveraging the concepts from this paper:

) Improvements and Capabilities of the Enhanced AI System (UniCross-Augmented System):

The enhanced system will be a single, distillation-trained cross-skill policy capable of performing long-horizon manipulation across diverse objects and hand morphologies without explicit skill switching.

) Specific Technical Improvements and Capabilities:

A single neural network policy, distilled from ten per-skill experts, ensures that the object state reached after a grasp is a valid starting point for relocation, and so on.

Capability: The system can autonomously execute complex sequences (e.g., Grasp + Relocate + Rotate) in one continuous run without requiring an explicit high-level planner to switch policies between tasks. This eliminates the failures common in systems that rely on sequential, skill-specific modules.

The system utilizes an interaction-aware object representation and a relational objective vector that dynamically tracks contact states and local geometric features, rather than relying on fixed shape priors or pre-defined contact regions.

Capability: The AI can manipulate objects of novel geometries (e.g., non-convex shapes, varying aspect ratios) and different scales (e.g., small wrappable objects vs. long elongated prisms) with high success rates, as demonstrated by its performance on unseen test sets in the experiments.

By sharing a unified observation space and action space, the policy is trained once and can be distilled for different hand morphologies (e.g., MANO vs. Sharpa Wave).

Capability: The system can operate effectively across a wide variety of robotic hand designs without needing to retrain or modify the core policy architecture for each new hardware embodiment, drastically reducing development time and enabling rapid deployment on diverse platforms.

The reward function explicitly includes a drop penalty and the training incorporates domain randomization (random force magnitudes, friction coefficients, and joint noise).

Capability: The AI can maintain stable object contact even when subjected to unexpected external forces or environmental perturbations (e.g., air gusts, bumps), significantly increasing the reliability of manipulation in real-world robotic applications.

The formulation explicitly models the four canonical skills (grasping, relocation, in-hand rotation, and translation) as instantiations of a shared task formulation.

Capability: The AI can perform multi-stage tasks where intermediate states are crucial for success (e.g., moving an object to a precise location before performing an in-hand reorientation), ensuring the overall mission objective is met through coordinated, coherent motion primitives rather than disconnected actions.