DeVI: Physics-based Dexterous Human-Object Interaction via Synthetic Video Imitation

summary

The gist

The gist: DeVI is a novel framework that leverages text-conditioned synthetic videos to enable physically plausible dexterous agent control for interacting with unseen target objects without

In short

DeVI learns physics-based control for dexterous human-object interaction by imitating synthetic videos generated from text prompts. It solves issues with unreliable 3D motion reconstruction by using a hybrid imitation target combining 3D human tracking and 2D object trajectories. This framework enables physically plausible manipulation without needing real motion capture data.

Key concepts

Synthetic Video Generation: A video diffusion model is used to create realistic, text-prompt-conditioned videos of human-object interactions. This allows the system to generate diverse scenarios and motions for objects and environments that were never actually recorded in the real world, serving as the primary training data source.
Hybrid Imitation Target: This novel target combines two references: a coarse 3D human reference derived from mesh recovery and a robust 2D object reference from tracked trajectories in the video. This combination helps overcome the imprecision of using only one type of tracking data, leading to more accurate imitation targets for physical control.
Visual HOI Alignment: This refinement step optimizes the coarse 3D human reference by aligning it against both the generated interaction video and the initial 3D object configuration. This ensures the learned human motion is physically realistic and properly positioned to interact with the target object during training.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Federated Learning — explained in 17 episodes
Retrieval-Augmented Generation (RAG) — explained in 16 episodes
Interpretability — explained in 15 episodes
Quantum Fisher Information (QFI) — explained in 15 episodes
Sample Complexity — explained in 15 episodes

This episode discusses

The paper

DeVI: Physics-based Dexterous Human-Object Interaction via Synthetic Video Imitation · Read on arXiv

Hyeonwoo Kim, Jeonghwan Kim, Kyungwon Cho, Hanbyul Joo

Seoul National University

Physics-based character control for dexterous Human-Object Interaction (HOI) relies heavily on motion-capture data, which limits its applicability beyond captured scenarios. Meanwhile, video generative models can synthesize realistic HOI videos for unseen objects and scenarios directly from text prompts, but 3D interactions learned from such synthetic data remain kinematic without ensuring physical plausibility. Imitating these videos with physics-based control is a promising direction, yet unreliable object tracking and hand-object misalignment make it difficult to obtain accurate imitation targets. We present DeVI (Dexterous Video Imitation), a novel framework that learns physics-based character control for dexterous HOI by imitating text-conditioned synthetic videos. To address unreliable 3D object motion reconstruction, we introduce hybrid imitation targets that combine reconstructed 3D human motion with tracked 2D object trajectories, bypassing the explicit reconstruction of the 3D object motion. Our Visual HOI Alignment further refines the human reference to align with both the generated video and the initial 3D object configuration, making it suitable for physical interaction. Extensive experiments show that DeVI's hybrid representation is more effective than those used by the baselines and that Visual HOI Alignment improves reconstruction quality and imitation success. We further demonstrate physically plausible functional manipulation from text instructions, including target-aware interactions and articulated object manipulation, without requiring motion-capture demonstrations.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "DeVI: Physics-based Dexterous Human-Object Interaction via Synthetic Video Imitation".

Tom: The gist:

Jane: First, who's behind it and why it matters.

Title and authors: Tom: So we're talking about this paper called DeVI, which stands for Dexterous Video Imitation. It’s all about using videos of human interactions with objects to teach an AI how to do those same things in a physics simulation, without needing those expensive three dee motion capture videos <ref:2604.20841#pg1>.

Jane: That’s right, Tom. The authors are looking at how we can use synthetic video data to give agents more realistic hands and movements when they have to grab or manipulate things in a virtual world.

Lu: It's really interesting because the paper tackles the problem of getting physically plausible human-object interaction motions, which is tough because those videos are often just 2D and not fully grounded in physics <ref:2604.20841#pg1,physically plausible human-object interaction>.

Meng: So what’s the big idea here? How do they get from a video to a working agent that can actually interact realistically?

Tom: Well, DeVI uses a video diffusion model to act as an interaction-aware motion planner. This means the model takes a text prompt and generates these physically plausible motions, which is pretty cool because it’s guided by what we see in the synthetic videos.

Jane: But they know that just relying on the 2D cues from those generated videos isn't enough for precise control, so they introduce a hybrid tracking reward system to fix that imprecision <ref:2604.20841#pg1>.

Meng: A hybrid tracking reward? That sounds complicated. How do you combine different types of tracking to make the agent learn better?

Tom: They integrate three dee human tracking with robust 2D object tracking into this one reward function, which is key because it helps the system learn both where the hand should go in three dimensions and how to keep that object in view on a flat plane <ref:2604.20841#pg1>.

Lu: And they use this hybrid approach to create a novel hybrid imitation target, combining a three dee human reference with a 2D object reference to guide the agent’s learning process <ref:2604.20841#pg1>.

Jane: That means the agent isn't just trying to copy one thing; it’s imitating both where the person is in space and what the object looks like in two dimensions at the same time.

Tom: The three dee human reference comes from recovering a human mesh along with a hand pose estimator, which they then optimize using visual HOI alignment, which refines that three dee reference by aligning it with both the video and the initial object state <ref:2604.20841#pg1>.

Meng: So there’s this whole process where they try to make sure the learned human motion matches what’s seen in the video while also being physically realistic for interacting with a specific three dee object <ref:2604.20841#pg1>.

Jane: Exactly, and that alignment is super important because it reduces pixel error, especially for those hand joints, which directly improves how close the agent gets to the actual three dee object at the moment of contact <ref:2604.20841#pg1>.

Title and authors: Tom: They define this hybrid tracking reward as R(st, at, gt) equals Rh times Ro times Rcontact. That’s where you get your three components working together in Reinforcement Learning to guide the humanoid control policy.

Lu: The human tracking reward, Rh, is designed to make the humanoid track that synthetic three dee human motion h-hat through joint difference rewards and a power penalty <ref:2604.20841#pg1>.

Meng: And then there’s Ro, the object tracking reward, which encourages the 2D projection of the simulated object to follow a reference 2D object trajectory o-hat, defined as e to lambda o times x hat minus x t squared <ref:2604.20841#pg1>.

Jane: That Ro term is essentially penalizing any deviation between where the agent thinks it should be and where the object looks in two dimensions over time.

Tom: And then there’s Rcontact, the contact reward, which they modulate using a binary contact label psi t inferred from 2D object point motion in the generated video V, which lets the humanoid learn how to actually establish contact with the target object <ref:2604.20841#pg1>.

Lu: The visual HOI alignment part is crucial because it minimizes two losses: Lb and Lh for the 2D body and hand projections, and they also use an HOI loss LHOI to enforce that human body parts are in contact with the object at a specific time instance <ref:2604.20841#pg1>.

Meng: So if I’m looking at this from an engineering standpoint, it sounds like they are using these multiple alignment losses to force the model to produce motions that look right and feel right when interacting physically.

Jane: It really is about making sure the human motion they generate is not just visually similar to the video, but actually feasible for physical interaction with the three dee object <ref:2604.20841#pg1>.

Tom: They've shown this works by training an actor-critic network where the actor uses a sequence transformer encoder to output actions and the critic estimates the value function based on those complex inputs.

Lu: The paper also showed that this framework can generalize beyond just tabletop scenarios, applying it to things like pushing or pick and place in non-tabletop environments.

Meng: That’s a big practical implication for robotics, moving these kinds of dexterous skills out of the lab bench and into more complex real-world settings.

Jane: Before we wrap up this discussion on DeVI, Tom and I want to touch on what the paper actually achieves overall, what it means for us who are just listening.

Tom: So in short, DeVI is a framework that imitates synthetic videos to control a physics-based character performing dexterous HOI in a zero-shot manner without needing high-quality three dee motion capture data <ref:2604.20841#pg1>.

Lu: It really shows how we can leverage video generation models to guide agents toward complex manipulation tasks that were previously only possible with expensive physical demonstrations.

Title and authors: Meng: From an engineering side, the fact that they can use a hybrid tracking reward built on top of 2D tracking rewards makes it efficient because getting 2D object trajectories is easier than acquiring six-dimensional poses for imitation <ref:2604.20841#pg1>.

Jane: And the visual HOI alignment is critical because it fine-tunes the motion, reducing pixel error and making the agent much more capable of achieving contact with a target object in three dimensions.

Tom: So to recap, DeVI introduces a hybrid imitation target combining three dee human and 2D object references, trained via a hybrid tracking reward that includes contact timing cues derived from video movement <ref:2604.20841#pg1>.

Lu: And the quantitative results show it outperforms baselines on the GRAB dataset for metrics like MPJPE (All) being less than zero point one, Tobj under zero point two, and Oobj under one point zero, which indicates it imitates reference motion very closely compared to other methods.

Meng: That success ratio of zero point five for MPJPE is significant because it shows the method is imitating the reference motion much more similarly than previous approaches we've seen for manipulation tasks like this on the GRAB dataset.

Jane: But they also noted that even when relying on relatively sparse 2D object trajectories as references, their success rate is higher compared to methods using six-dimensional poses <ref:2604.20841#pg1>.

Tom: They also pointed out that without visual HOI alignment, if you just estimate the body and hand poses independently, they get unified into a single SMPL-X model which causes severe misalignment with both the video and the three dee object <ref:2604.20841#pg1>.

Lu: That's a strong point, because it shows that aligning those references is not optional; it’s essential for producing motion that works in the physical world.

Meng: And their study on limitations also points out that they might have some estimation errors because they rely solely on pixel velocity of hands and objects, suggesting that affordance grounding methods could further refine those contact labels.

Jane: So, while DeVI is a strong framework for imitation, it’s important to remember that the reliance on visual cues means we still have to be careful about how well those cues are grounded in reality.

Tom: This paper, DeVI: Physics-based Dexterous Human-Object Interaction via Synthetic Video Imitation, gives us a solid foundation for using synthetic video data to train agents for dexterous manipulation without needing tons of three dee mocap data <ref:2604.20841#pg1,DeVI: Physics-based Dexterous Human-Object Interaction via Synthetic Video Imitation>.

Lu: It opens up a way for us to explore complex interactions across many different objects and scenarios that we could never capture with traditional methods.

Meng: For practical deployment, the ability to push these skills into non-tabletop environments, like pick and place tasks, is what makes this research really relevant for robotics applications.

Jane: That’s a good summary of where DeVI sits right now—a powerful tool that bridges the gap between realistic video generation and physical agent control through careful reward design.

The paper's summary: Tom: So, we've seen how DeVI uses those synthetic videos to teach an AI how to grab and move things in a virtual world without needing those expensive motion capture videos.

Jane: Exactly, it’s about taking what we see in these generated videos and turning it into real control for a physics-based character.

Tom: The core idea is using the video diffusion model as a way to plan the movement, which gives us this physically plausible interaction motion based on a text prompt.

Lu: And to make sure that 2D video isn't enough, they put in this hybrid tracking reward system that mixes three dee human tracking with 2D object tracking <ref:2604.20841#pg1>.

Meng: So if you're driving or cooking and you see a video of someone picking up an apple, this method is trying to learn how to do that by looking at both the person's three dee position and the apple's movement on a screen.

Jane: Right. And they use this hybrid imitation target, which combines a three dee human reference with a 2D object reference to guide the training of the agent’s policy <ref:2604.20841#pg1>.

Tom: That means when you train the AI, it’s not just copying one thing; it has to figure out how to imitate both signals simultaneously.

Lu: The three dee human reference comes from recovering a human mesh and using a hand pose estimator, which they then refine with visual HOI alignment.

Jane: That alignment part is super important because it fixes the errors, especially for the hand joints, making sure the motion looks right when interacting with a real three dee object.

Meng: And they define this reward function as Rh times Ro times Rcontact. So you get rewards for tracking the human, tracking the object's 2D position, and actually making contact <ref:2604.20841#pg1>.

Tom: The contact reward is special because it’s based on inferring when contact happens from the video itself, which helps the AI learn how to physically touch something.

Lu: The results are pretty solid on datasets like GRAB. They show a success ratio of zero point five hundred for MPJPE less than zero point one which means it’s imitating the reference motion really well compared to other methods we’ve seen.

Jane: And they found that even when they rely on just these sparse 2D object trajectories as references, their success rate is higher than methods using six-dimensional poses <ref:2604.20841#pg1>.

Meng: But there's a catch, and this is something you need to know. They showed that without that visual HOI alignment, if you just estimate the body and hand poses separately, they all get lumped into one SMPL-X model, which causes bad alignment with the video and the object.

Tom: So it really proves that aligning those references isn't optional; it's essential for making sure what the AI learns is actually feasible in a physical world.

Lu: They also mentioned that because they rely on pixel velocity of hands and objects, there might be some estimation errors, suggesting we need more methods to ground those contact labels better.

Jane: It’s a strong framework for using video generation to train agents for complex manipulation without needing tons of three dee motion capture data.

Tom: This opens up a way to explore many different objects and scenarios that we could never capture with traditional methods, which is pretty cool for robotics applications.

Lu: And because they can apply this beyond tabletop settings, like pushing or pick and place in non-tabletop environments, it’s really relevant for real-world deployment.

Meng: So DeVI gives us a solid foundation to bridge the gap between realistic video generation and physical agent control through careful reward design.

The paper's improvements: Tom: So, we've talked about how DeVI uses those synthetic videos to teach an AI how to grab and move things in a virtual world without needing those expensive motion capture videos.

Jane: It’s about taking what we see in these generated videos and turning it into real control for a physics-based character.

Tom: The core idea is using the video diffusion model as a way to plan the movement, which gives us this physically plausible interaction motion based on a text prompt.

Lu: And to make sure that 2D video isn't enough, they put in this hybrid tracking reward system that mixes three dee human tracking with 2D object tracking <ref:2604.20841#pg1>.

Jane: Right. And they use this hybrid imitation target, which combines a three dee human reference with a 2D object reference to guide the training of the agent’s policy <ref:2604.20841#pg1>.

Tom: That means when you train the AI, it’s not just copying one thing; it has to figure out how to imitate both signals simultaneously.

Lu: The three dee human reference comes from recovering a human mesh and using a hand pose estimator, which they then refine with visual HOI alignment.

Jane: That alignment part is super important because it fixes the errors, especially for the hand joints, making sure the motion looks right when interacting with a real three dee object.

Tom: The contact reward is special because it’s based on inferring when contact happens from the video itself, which helps the AI learn how to physically touch something.

Lu: They also pointed out that without that visual HOI alignment, if you just estimate the body and hand poses separately, they all get lumped into one SMPL-X model, which causes bad alignment with the video and the object.

Jane: It really proves that aligning those references isn't optional; it's essential for making sure what the AI learns is actually feasible in a physical world.

Tom: And they also mentioned that because they rely on pixel velocity of hands and objects, there might be some estimation errors, suggesting we need more methods to ground those contact labels better.

Lu: The authors suggest that if we want to push this further, we should look into affordance grounding methods to refine those contact labels.

Meng: That makes sense. From an engineering standpoint, they're pointing out the next logical step is making sure the AI understands what it's touching in a more intuitive way than just looking at pixels.

Jane: It’s about moving from just imitation to true physical understanding, where the agent knows *why* it should touch something in a certain way.

Tom: So they’re not stopping there; they’re suggesting that we need better ways to connect those visual cues to real-world physics.

Lu: Think about what this means for culture—if we can build agents that understand interaction through these synthetic videos, it opens up huge possibilities for how we design interactive systems.

Jane: It means the AI learns more than just a trick; it starts building a model of physical interaction based on visual examples.

Tom: And that leads us into what they might do next, exploring those affordance grounding ideas to make the system even more robust when encountering new objects or environments.

Conclusion: Tom: So we’ve covered how DeVI uses video diffusion to teach AI dexterous human-object interaction without needing expensive motion capture data, and its success on datasets like GRAB is quite impressive at zero point five hundred for MPJPE less than zero point one.

Jane: It really shows how we can use generative video models to guide agents toward complex manipulation tasks that were previously only possible with those expensive physical demonstrations.