Learning to Generate Rigid Body Interactions with Video Diffusion Models

arXiv:2510.02284 · cs.CV, cs.AI, cs.LG · Submitted 2025-10-02 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Learning to Generate Rigid Body Interactions with Video Diffusion Models".

Tom: Recent video generation models struggle to produce physically plausible object interactions and lack object-level control mechanisms, which limits their utility as world simulators for robotics.

Jane: First, who's behind it and why it matters.

Title and authors: Tom: So, let's talk about the title and who wrote this paper: "Learning to Generate Rigid Body Interactions with Video Diffusion Models." It immediately tells us that the core problem they are solving is making sure objects in generated videos behave like real-world physical bodies.

Jane: And the authors—David Romero, Ariana Bermudez, Viacheslav Iablochnikov, Hao Li, Fabio Pizzati, and Ivan Laptev—they are researchers from MBZUAI and Picscreen. It sounds like a solid group of experts tackling this complex problem head-on.

Lu: They're tackling the inherent weakness in current video diffusion models where they struggle to generate physically plausible interactions and lack object-level control mechanisms, which limits their use as world simulators for robotics.

Meng: So, what does this title really mean for practical engineering? It means we're moving past just pretty visuals to something that has verifiable physical dynamics, which is crucial if we want these models to be reliable tools for training robots in complex environments.

Lalam: I see the authors are focused on bridging the gap between visual generation and physical simulation; it suggests they are trying to make AI systems that can actually *act* in a physically grounded way.

The paper's summary: Tom: Moving on to what KineMask actually does, the paper summarizes their approach as introducing a two-stage training strategy that uses object masks to gradually remove future motion supervision, allowing the video diffusion models to learn how to synthesize complex dynamics starting only from initial conditions.

Jane: That’s a clever way they’re training it; instead of showing the model everything at once, they let it learn by progressively taking away the guidance about what happens next, focusing on generating motion purely from where things start.

Lu: The paper also shows how this method enables object-based control by encoding instantaneous velocity as a mask and combining that low-level kinematic control with high-level textual conditioning using a Vision Language Model like GPT for inference.

Meng: So, they’re not just giving the model raw motion commands; they're giving it a structured way to control object direction and speed through masks, which is much more manageable for an AI system to process during generation.

Lalam: It seems like the summary emphasizes that KineMask successfully trains pretrained video diffusion models to synthesize realistic rigid body interactions in real-world input scenes, addressing the struggle current models have with things like object permanence and collisions.

The paper's improvements: Tom: The main improvement they highlight is that KineMask outperforms state-of-the-art models of comparable size on motion fidelity, interaction quality, and overall physical consistency when tested on real scenes. They also show that the importance of both the proposed training strategy and the integration of low- and high-level controls is significant.

Jane: That's a strong claim, showing that their method isn't just tweaking parameters; it genuinely improves how well these models handle complex physical scenarios compared to what we see in existing research.

Lu: A key improvement they demonstrate is the generalization of KineMask across different Video Diffusion Models, showing improvements when applied to CogVideoX-5B, Wan2 point 2-5B, and Cosmos2 point 5-2B.

Meng: That generalization is something I’m really interested in because it means the technique isn't tied to one specific model architecture; it suggests the underlying mechanism for physical control is robust enough to work across different setups.

Lalam: The authors also show that training on a specific "Interactions" dataset significantly boosts performance compared to training only on a "Simple Motion" dataset, proving that using the right data is essential for rendering complex object interactions.

Conclusion: Tom: So, wrapping up the paper "Learning to Generate Rigid Body Interactions with Video Diffusion Models," they show KineMask effectively generates realistic multi-object interactions while giving control over variable object velocities and proves its applicability across different video diffusion models.

Jane: It sounds like the authors have successfully shown that combining low-level motion control with high-level textual conditioning is a powerful way to ground these generative models in physical reality for the first time.

Lu: The implications are significant because it shows a path forward for establishing video diffusion models as reliable world simulators, which is a huge step toward building robust robotics.

Meng: From an engineering standpoint, this means we can start designing AI agents that rely on these generated environments knowing the physics are consistently applied during training and testing.

Lalam: It really shows that by thoughtfully structuring the training and conditioning, we can push generative models beyond just looking real to actually modeling causal physical effects in synthesized scenes.

MBZUAI

cs.CV, cs.AI, cs.LG

Submitted: 2025-10-02

Updated: 2026-09-27

Project page: https://daromog.github.io/KineMask

Importance score: 89/100

The gist: Recent video generation models struggle to produce physically plausible object interactions and lack object-level control mechanisms, which limits their utility as world simulators for robotics.

Key concepts

KineMask Framework
A two-stage training strategy designed to teach video diffusion models how to create realistic object interactions. It uses velocity masks for low-level control and textual descriptions for high-level guidance, allowing the model to synthesize complex dynamics from simple initial conditions.
Low-Level Motion Control via Velocity Masks
This technique encodes an object's instantaneous speed and direction as a mask. In the training process, this mask guides a ControlNet to structure the motion in generated videos. Later, dropping parts of these masks forces the model to learn dynamics purely from starting conditions.
High-Level Textual Conditioning
This involves using text prompts describing future scene dynamics alongside visual input. During inference, a Vision-Language Model (VLM) interprets the prompt and infers high-level outcomes, such as liquid spilling after a collision, which is then combined with the low-level control signal.
Two-Stage Training Strategy
The training process involves two distinct phases. The first phase trains a ControlNet using aggregated velocity masks from simulator data. The second phase employs mask dropout to train the VDM to synthesize motion starting only from initial conditions, conditioned on the learned controls.

Terminology

Summary

Recent video generation models struggle to produce physically plausible object interactions and lack object-level control mechanisms, which limits their utility as world simulators for robotics. This paper introduces KineMask, a framework designed to enable video diffusion models (VDMs) to generate realistic rigid body control, interactions, and effects by controlling initial object velocity and integrating low-level kinematic control with high-level textual conditioning.

KineMask Framework Overview

The core of KineMask is a two-stage training strategy that gradually removes future motion supervision via object masks to train video diffusion models (VDMs) on synthetic scenes of simple interactions. This strategy allows the model to learn how to synthesize complex dynamics starting only from initial conditions. The framework aims to answer two central questions for world models: (1) Can a video diffusion model generate realistic interactions between rigid bodies given initial dynamic conditions?, and (2) how do data and textual conditioning influence the emergence of causal physical effects in generated videos?

Low-Level Motion Control via Velocity Masks

The method enables low-level kinematic control over parameters such as object direction and speed by encoding the object's instantaneous velocity as a mask. This is achieved through a novel training procedure:

  1. In the first stage, KineMask trains a ControlNet to map dense pixel-wise supervision into structured guidance for object motion in generated videos, using aggregated velocity masks derived from simulator-rendered videos with explicit ground-truth dynamics.

  2. In the second stage, a mask dropout strategy is employed during training by randomly erasing parts of the velocity masks, leading to a truncated mask tensor where only some frames contain velocity supervision. This forces the VDM to synthesize motion dynamics starting from initial conditions only, conditioned on the object's initial velocity and inferred high-level outcomes.

High-Level Textual Conditioning

Beyond low-level control, KineMask integrates high-level prompt conditioning through textual descriptions of future scene dynamics. During inference, the model constructs the low-level conditioning with a SAM mask for an unseen image and uses a Vision Language Model (VLM) like GPT to infer high-level outcomes of object motion from the single input frame. This allows for the synthesis of complex effects, such as liquid spilling as a result of a collision, by combining this textual description with the low-level control signal.

Training Data and Generalization

KineMask is trained on synthetic data constructed in Blender, where scenes are rendered with boxes and cylinders placed on textured surfaces. The training pipeline requires both low-level conditioning (aggregated velocity masks) and high-level conditioning (textual descriptions). To generate the textual descriptions, a Vision–Language Model (VLM) is prompted to provide detailed video captions focusing on object interactions. Furthermore, the method demonstrates generalization across different Video Diffusion Models (VDMs), showing improvements when applied to models such as CogVideoX-5B, Wan2.2-5B, and Cosmos2.5-2B.

Experimental Results and Contributions

Experiments show that KineMask outperforms state-of-the-art models of comparable size on motion fidelity, interaction quality, and overall physical consistency when tested on real scenes. Ablation studies highlight the importance of both the proposed two-stage training strategy and the integration of low- and high-level controls. Specifically, training on an Interactions dataset significantly boosts performance compared to training only on a Simple Motion dataset, demonstrating that KineMask trained on appropriate data allows it to render complex object interactions. The final results show that KineMask synthesizes realistic multi-object interactions while allowing control over variable object velocities and demonstrates its applicability and generalization to different VDMs.

The gist: KineMask is a framework for generating accurate object interactions and effects in complex scenes by enabling video diffusion models to generate rigid body control by controlling initial object velocity only, trained via a novel two-stage training strategy that integrates low-level kinematic control with high-level textual conditioning.

Limitations and Future Directions

A limitation noted is that the low-level conditioning is limited to velocity, whereas real-world motion also depends on factors such as friction, mass, shape, and air resistance. Incorporating these controls is a promising direction for making VDM-generated motion more physically accurate. Moreover, extending to soft-body interactions will further improve world modeling capabilities. The paper concludes that combining text-based conditioning with KineMask enhances realism and supports the joint use of both modalities for physically grounded world modeling.

Key Contributions Enumerated:

  1. Introduction of KineMask, a mechanism for object motion conditioning in VDMs based on a novel two-stage training and conditioning encoding.

  2. Training KineMask on a synthetic video dataset composed of simple object interactions and demonstrating resulting models to enable generation of complex object interactions in realistic scenes.

Improvements for AI systems

Here are the specific improvements that can be made to existing AI systems, based on the KineMask framework described in this paper:

The core improvements revolve around moving Video Diffusion Models (VDMs) from merely synthesizing visually pleasing, temporally consistent video frames to generating videos with verifiable physical plausibility and precise object-level control.

Specific improvements and capabilities include:

  1. Enhancing Robotics and Embodied Decision Making through Physically Plausible World Models:

  2. Enabling Fine-Grained, Object-Level Control in Video Generation:

  3. Improving Causal Understanding in Generated Dynamics (Collision and Interaction Synthesis):

  4. Facilitating Generalization to Diverse Video Diffusion Model Architectures:

Detailed breakdown of what the improved AI system can do:

  1. The improved system can serve as a more reliable World Simulator for robotics and embodied decision-making by generating videos where object interactions (collisions, pushes) adhere strictly to rigid body physics. This allows robots trained on these generated environments to operate with higher confidence, knowing that the simulated dynamics respect fundamental laws of motion (e.g., mass awareness during collisions).

  2. The system can enable precise control over object kinematics by allowing users or robotic systems to specify desired initial object velocities and directions via a mask encoding. This moves beyond coarse textual prompts to offer low-level kinematic control, allowing for scenarios where an agent needs to initiate motion with specific speeds and trajectories, such as pushing an object at a precise velocity.

  3. The system can synthesize complex, causal physical effects that are difficult for current models to capture on their own. Specifically, it can generate realistic consequences of interactions like:

@ Collision Synthesis: Generating accurate collisions between multiple objects, including the correct resulting motion and displacement of both colliding bodies (e.g., one object being pushed or knocked over).

@ Environmental Effects: Synthesizing secondary physical phenomena resulting from interactions, such as liquid spilling due to a collision or water ripples spreading outward from an object's movement.

@ Mass Awareness: Demonstrating understanding of mass during collisions by accurately predicting the final position and velocity changes based on the relative masses of interacting objects.

  1. The improved system exhibits strong generalization capabilities across different Video Diffusion Model (VDM) architectures (e.g., CogVideoX, Wan2.2-5B, Cosmos2.5-2B). This means a single training methodology can be applied to a wide variety of generative models, resulting in state-of-the-art performance on each backbone without requiring model retraining for every new architecture.

In summary, the improved AI system transitions from being a high-quality video generator to a physically grounded dynamic scene synthesizer capable of simulating complex physical interactions with user-defined motion inputs.

Sources

Related papers