Dex2HOI: Dexterous Bimanual Two-Object Interaction Generation

arXiv:2605.30444 · cs.CV · Submitted 2026-05-28 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Dex2HOI: Dexterous Bimanual Two-Object Interaction Generation".

Tom: Dex2HOI presents a unified diffusion model designed to synthesize dexterous bimanual manipulations involving up to two objects from text, addressing the research gap concerning coordinated multi-object interaction in human behavior.

Jane: First, who's behind it and why it matters.

Title and authors: Tom: So, we’re diving into the Dex2HOI paper today, which has the title Dexterous Bimanual Two-Object Interaction Generation. The authors are Chrysa Pratikaki, Pablo Ruiz-Ponce, Jiankang Deng, and Stefanos Zafeiriou. This whole setup is designed to fix a gap where current research mostly focuses on single-object manipulation.

Jane: Exactly; the core idea here is that human behavior naturally involves coordinating both hands to work with multiple objects simultaneously, which is something previous models hadn't fully explored. It moves beyond just one hand touching one item.

Lu: What caught my eye about the title is the emphasis on "Dexterous Bimanual," suggesting they are focusing on the fine motor skills and coordination aspect, not just basic object placement. It sounds like a deep dive into realistic human interaction dynamics.

Meng: I'm thinking about how they structure the problem—unifying single- and two-object synthesis into one model—that seems like a significant architectural choice for efficiency, though I need to see if that unification actually simplifies training or just complicates it.

Lalam: For me, the implication is that we can start generating much richer and more complex scenarios simply from a text prompt without needing separate pipelines for each object interaction.

The paper's summary: Tom: Now we look at the summary of Dex2HOI, and it seems the authors are proposing a Dual-Stream Diffusion approach. They treat each object as its own dedicated interaction stream, which is then coordinated using bidirectional cross-attention.

Jane: That dual-stream idea sounds clever because it lets the model process the motion related to object A and object B separately before bringing them together. It’s like having two specialized teams working on the same task but communicating constantly.

Lu: The paper introduces a novel hand-relative object representation, defining features like global rotation and translation relative to the wrists, which helps disentangle motion into wrist-local and global branches. That’s a really insightful way to represent physical interaction from a human perspective.

Meng: I'm interested in those learned mixture weights that adapt based on whether a hand is in contact with an object, which allows for smooth interpolation between left-hand and right-hand control. That kind of dynamic weighting sounds promising for handling real-world variability.

Lalam: From a cultural standpoint, this means our AI can start modeling more complex social or physical interactions that involve multiple tools or items in a single scene, making simulations much more grounded.

The paper's improvements: Tom: Regarding the improvements they suggest, Dex2HOI focuses on unifying the framework, which means it supports both single-object and two-object HOI synthesis without needing separate models. That’s a big simplification for researchers trying to build general tools.

Jane: And they also mention synthesizing physically plausible, contact-aware motions in just one diffusion sampling step, which avoids the need for slower test-time optimization methods like DNO. That speed advantage is quite something for deployment.

Lu: The paper details how they use a Motion Fusion Network that operates only on the human representations to merge the outputs from the two streams into a single coherent pose. It seems like this network is crucial for ensuring the final motion looks like it actually makes sense physically.

Meng: That fusion step sounds computationally intensive, though if it achieves that high degree of physical plausibility in one go, it could reduce the complexity of downstream tasks that need to refine the motion later. I just want to know how scalable this fusion network is for very large models.

Lalam: The paper also includes several contact-aware supervision terms in their training objective, such as supervising the mixture weights and object translation against ground-truth world-space positions. This suggests they are really prioritizing the physical accuracy of the interaction itself.

Conclusion: Tom: So, to wrap up on Dex2HOI, it’s a unified diffusion model that uses dual streams and motion fusion to generate dexterous bimanual two-object interactions from text prompts. The results show state-of-the-art performance for both single and two-object manipulation in a single shot.

Jane: It’s really about bringing coordinated multi-object interaction, which we thought was hard to model, into the realm of practical generation using diffusion. This work provides a solid foundation for simulating complex physical coordination in AI systems.

Lu: The implications for creating embodied agents or realistic robotics that can handle cluttered environments with two objects simultaneously are quite significant; it opens up new possibilities for how we program physical interaction.

Meng: From an engineering perspective, the unification and the single-shot synthesis capability could drastically simplify the development cycle for applications that require complex manipulation, reducing reliance on multi-stage pipelines.

Lalam: I think what this paper really shows us is that by focusing on how objects are represented relative to the hands, we can build representations that capture the physics of coordination much better, which will help make our AI agents feel more intuitive and capable.

Tom: It’s been fascinating exploring Dex2HOI today. We’ll be keeping a close eye on how this unified model evolves as researchers continue to build on this foundation.

Jane: Indeed, it’s clear that the focus on contact-aware supervision is going to be key for moving these synthesized motions into real-world applications soon.

Lu: We look forward to seeing how future work expands upon the representation of object motion described in page zero of Dex2HOI: Dexterous Bimanual Two-Object Interaction Generation.

Chrysa Pratikaki, Pablo Ruiz-Ponce, Jiankang Deng, Stefanos Zafeiriou, Rolandos Alexandros Potamias

Imperial College London · University of Alicante

cs.CV

Submitted: 2026-05-28

Updated: 2026-09-28

Importance score: 89/100

The gist: Dex2HOI presents a unified diffusion model designed to synthesize dexterous bimanual manipulations involving up to two objects from text, addressing the research gap concerning coordinated

Key concepts

Hand-Relative Object Representation
This novel representation breaks down object motion into wrist-local and global components. It defines object features based on rotation and translation relative to the left and right wrists, allowing the model to understand how objects move specifically from a hand's perspective rather than just globally.
Dual-Stream Diffusion Approach
The model processes each object in a separate stream (A for object A, B for object B). These streams share human motion data but handle their respective objects differently. They coordinate by applying bidirectional cross-attention after every layer, allowing the streams to effectively 'attend' to each other's information.
Motion Fusion Network (Dh)
This network takes the separate predictions from the two object streams and reconciles them into a single, coherent human pose. It operates only on human representations and uses self-attention layers to merge the data from both streams into one unified output for smooth, realistic bimanual movement.

Terminology

Summary

Dex2HOI presents a unified diffusion model designed to synthesize dexterous bimanual manipulations involving up to two objects from text, addressing the research gap concerning coordinated multi-object interaction in human behavior. The core contribution is a framework that unifies single- and two-object HOI synthesis by employing a Dual-Stream Diffusion approach coordinated via bidirectional cross-attention and fusing outputs through a Motion Fusion Network, achieving state-of-the-art results for both single and two-object manipulation in a single shot.

The Gist

Dex2HOI is a unified diffusion model for Human-Object Interaction that generates dexterous bimanual manipulations from text, supporting simultaneous interaction with up to two objects.

How it works: Representation and Decompositions

The model introduces a novel hand-relative object representation to disentangle object motion into wrist-local and global branches, moving beyond relying solely on global trajectories. For a sequence of length T, the full motion feature at frame t is defined as:

ft = h t, r t, o A t, o B t (1)

The object feature vector is defined as:

o obj t = [r obj t, ∆L t, ∆R t, pG t, αT] (2)

where the components include global object rotation and translation in the left and right wrist frames. The model defines a global object translation through a soft mixture of these relative representations:

pˆ obj t = w L t p L t + RL t ∆L t + w R t p R t + RR t ∆R T + w G T pG T (4)

The learned mixture weights [w L, wR, wG] are determined by softmax(αT), allowing the model to smoothly interpolate between left-hand control, right-hand control, and global motion through the learned weights [w L t, w R t, w G t], which adapt at each frame depending on which hand is in contact with the object.

How it works: Dual-Stream Diffusion and Coordination

Dex2HOI employs a Dual-Stream Diffusion approach where each object is processed in a dedicated interaction stream and coordinated through bidirectional cross-attention.

  1. Stream A carries motion representation f A t = [h t, r t, o A t], and Stream B carries f B t = [h t, r t, o B T]. Both streams share the human representation [h t, r T] but differ in their object representations.

  2. At diffusion timestep τ, the shared denoiser Dθ processes each stream conditioned on the other:

ˆf A 0 = Dθ f A τ, f B τ, c A, τ (5)

ˆf B 0 = Dθ f B τ, f A T, c B, τ (5)

  1. Information flow between the two HOI streams is achieved by applying bidirectional cross-attention CA(·) after each layer l of the shared HOI Diffusion Dθ, resulting in each stream effectively attending to the other.

How it works: Motion Fusion and Training Objective

The outputs from the two streams are reconciled using a dedicated Motion Fusion Network Dh.

  1. Both streams produce complete predictions, which are decomposed along the feature partition introduced in Section 3.1:

ˆf A 0 = [hˆ A, ˆr A, ô A], and ˆf B 0 = [ĥ B, ˆr B, ô B] (6)

  1. The Motion Fusion Network Dh is implemented as a stack of self-attention layers that operates only on the human representations to produce a single coherent human pose:

hˆ, ˆr = Dh(ĥ A, ˆr A, ĥ B, ˆr B) (7)

  1. The training objective L is comprehensive and includes the main diffusion objective Ldif f along with several contact-aware supervision terms:

L = Ldif f + Lvel + λwLmix + λtLtrans + λoLof object representation supervision +λpLdist (12)

This includes supervising the mixture weights (Lmix), the decoded object translation against ground-truth world-space position (Ltrans), and regularizing wrist-relative offsets against ground-truth values (oLof).

Improvements for AI systems

Here are the specific improvements that Dex2HOI enables in AI systems, and what those improved systems can achieve:

  1. The ability to generate high-fidelity, temporally coherent, full-body human motions involving simultaneous manipulation of two distinct objects from natural language text prompts (e.g., The subject reaches for and grasps the spatula and mixing bowl.).

  2. Transitioning HOI generation from a single-object capability to a unified framework supporting both single-object and bimanual (two-object) interactions without requiring separate models or multi-stage pipelines.

  3. Synthesizing physically plausible, contact-aware motions in a single diffusion sampling step, thereby eliminating the need for costly and slow test-time optimization (like DNO). This results in up to a ×540 inference speed-up over prior state-of-the-art methods.

  4. Achieving superior quantitative performance across diverse benchmarks, including state-of-the-art results on both single (GRAB) and two (HUMOTO/HIMO) object manipulation tasks, validated by metrics like FID, Diversity, and Action Retrieval.

  5. Producing motions that exhibit high perceptual realism and accurate physical plausibility when assessed by advanced motion evaluators (FID), VLM assessments of naturalness/contact quality, and user preference studies.

  6. Enabling the creation of embodied agents (robotics) or realistic simulations that can execute complex, coordinated bimanual tasks in cluttered environments with two objects simultaneously, guided by high-level text instructions.

Sources

Related papers