Tri-Prompting: Video Diffusion with Unified Control over Scene, Subject, and Motion

arXiv:2603.15614 · cs.CV · Submitted 2026-03-16 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Tri-Prompting: Video Diffusion with Unified Control over Scene, Subject, and Motion".

Jane: Tri-Prompting introduces a unified video diffusion framework that integrates scene composition, multi-view subject consistency, and disentangled motion control within a single model.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, wrapping up our discussion on Tri-Prompting, the authors are clearly showing how they’ve managed to unify scene composition, multi-view subject control, and disentangled motion control within one model. Jane It boils down to using a dual-conditioning signal—three dee tracking points for the background and downsampled RGB cues for the foreground—to drive this unified control <ref:2603.15614#pg1,and downsampled RGB cues for>.

Meng: The result is that they've managed to resolve coarse motion cues into high-fidelity, three dee-consistent subjects by using these two distinct conditioning inputs <ref:2603.15614#pg1>. Lu This dual conditioning naturally decouples the background and foreground motion, which is a significant technical achievement in maintaining spatial coherence during generation.

Lalam: I think the real impact here is how this framework supports novel workflows, specifically three dee-aware subject insertion into any scene and manipulation of existing subjects in an image <ref:2603.15614#pg1,and manipulation of existing subjects in an image>. Tom That capability means users can now control both the camera pose and the character motion in a natural way while keeping the appearance consistent across different views, which is really useful for creative applications.

Jane: And looking at the experimental results, Tri-Prompting significantly outperforms specialized baselines like Phantom and DaS on metrics like multi-view identity preservation and three dee consistency <ref:2603.15614#pg1,Tri-Prompting significantly outperforms specialized baselines>. Lu This suggests that the unified approach isn't just theoretically interesting; it delivers tangible improvements in how well these models capture the visual fidelity of subjects across various camera angles.

Tom: The authors are using a ControlNet scale schedule strategy during inference to help balance controllability with realism, which is a practical consideration for anyone deploying this kind of generation system. Meng It shows they thought about the user experience when setting up these complex controls; you don't want it to be completely locked down or overly constrained.

Lalam: The future work mentioned points toward making this framework even more accessible, suggesting that the ability to harmonize scene and subject control opens up new avenues for creating highly controlled, narrative video content in the near future.

Conclusion: Tom: So, we've been diving deep into Tri-Prompting, and now it’s time to wrap up what this paper is all about—looking at the title and who cooked this up. Jane, can you give us a quick rundown of what that title really means for someone listening who might not be an expert?

Jane: Absolutely, Tom. The core idea of "Tri-Prompting" is that it’s introducing a single diffusion framework designed to handle three things at once: scene composition, keeping the subject consistent across different views, and controlling how everything moves independently. It’s about getting the background, the person in it, and their movement all under one unified model.

Lu: What's particularly interesting is how they tackle that unity; they use a dual-conditioning motion module driven by three dee tracking points for the scene and downsampled RGB cues for the subject. That really suggests a smart way to separate those complex motion signals.

Meng: From an engineering standpoint, what I find compelling is that this unified approach manages to disentangle those motions naturally, which means we don't have to build three separate control systems just to get scene, subject, and movement together correctly in the video generation process.

Lalam: I think the implication here for culture is huge because if we can achieve this level of control over visual narrative—scene, character identity, and action—it opens up incredible possibilities for creating immersive storytelling experiences that are far more nuanced than what we see now.

Tom: Exactly! And looking at the authors, they’ve done some impressive work comparing their model against specialized baselines like Phantom and DaS. That comparison really hammers home just how much better Tri-Prompting is on metrics like motion accuracy and identity preservation.

Jane: It really does show a tangible improvement in how well the generated videos maintain consistency across different viewpoints, which is a big deal when you’re trying to make something look real. The authors seem very focused on proving that this unified method actually works better than existing specialized tools.

Lu: I'm especially excited about the potential for three dee-aware subject insertion they describe; that sounds like it could fundamentally change how we composite characters into pre-existing or generated environments in a realistic way.

Meng: That level of manipulation capability is interesting because it moves beyond simple generation; it lets users actively sculpt the scene by controlling both the camera and the subject's movement precisely. That’s a different kind of interaction than just typing a prompt.

Lalam: For culture, I see this translating into tools where creators can build complex digital worlds with highly controllable entities that interact realistically, which could influence how we consume media in the long run.

Tom: So, to sum it up for our listeners, Tri-Prompting is a model that combines scene setup and subject control into one system using clever dual conditioning signals to handle motion separately yet coherently. Where should we go next after talking about the mechanics?

Adobe Research

cs.CV

Submitted: 2026-03-16

Updated: 2026-10-06

Comments: Project page: https://zhouzhenghong-gt.github.io/Tri-Prompting-Page/

Project page: https://zhouzhenghong-gt.github.io/Tri-Prompting-Page/Multi

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 91/100

The gist: Tri-Prompting introduces a unified video diffusion framework that integrates scene composition, multi-view subject consistency, and disentangled motion control within a single model.

Key concepts

Dual-Conditioning Motion Control
This method uses two distinct signals to guide motion control. One signal, derived from 3D tracking points, controls the background scene's movement. The second signal, a low-resolution RGB proxy of the subject, controls how the foreground character moves independently. This separation allows for fine-grained control over both elements.
Scene and Subject Latent Prepending
During training, the model is conditioned on three latent components: the scene latent ($z_I$), multi-view subject latents ($z_V$), and a subject latent ($z_S$). By prepending the scene latent to the sequence, the framework establishes foundational control over where and who exists in the video before motion is applied.
ControlNet Scale Schedule Strategy
During inference, this strategy manages how strongly the motion control signal influences generation. The strength of this signal (the scale 's') is gradually reduced during initial steps. This prevents the model from being overly constrained too early, ensuring that generated gaits look smooth and realistic rather than stiff or unnatural.

Terminology

Summary

Tri-Prompting introduces a unified video diffusion framework that integrates scene composition, multi-view subject consistency, and disentangled motion control within a single model. The core contribution is a dual-conditioning motion module driven by 3D tracking points for background scenes and downsampled RGB cues for foreground subjects, enabling precise control over where the story happens (scene), who is in it (subject), and how they move (motion). This unified approach significantly outperforms specialized baselines like Phantom and DaS across motion accuracy, multi-view identity preservation, and 3D consistency metrics.

How it works

The framework leverages a two-stage training paradigm to achieve comprehensive control. Stage 1 focuses on establishing foundational control by encoding the first-frame image (scene) and multi-view subject images into latent components, specifically prepending the scene latent and appending the subject latent to form the input token sequence: z seq ← [z I, z V, z S]. This stage utilizes LoRA to condition on these inputs for first-frame generation and cross-view identity. Stage 2 involves training a ControlNet module to incorporate dual-conditioning motion control, where the model is conditioned on both scene and subject motion signals.

Dual-Conditioning Motion Control

The paper defines a dual-conditioning signal, including a 3D tracking point for background, and downsampled RGB for foreground. For scene (background) control, it follows DaS to construct an XYZ tracking point video where coordinates are determined by position and depth in the first frame. For subject (foreground) control, it uses a low-resolution downsampled RGB point proxy Msubject obtained by downsampling the subject pixels into a fixed grid (e.g., 70 × 70) within the subject region. These two signals are composite into an anchor motion control video in a spatially exclusive manner. The ControlNet module then uses these dual cues to update the video latents: z V ← z V + s · ControlNet([z I, z M, z S])[1:1 + T/tc].

Training and Inference Strategy

Tri-Prompting employs a two-stage training strategy. Stage 1 fine-tunes the base model with LoRA on attention and MLP blocks to establish scene and subject control. Stage 2 freezes the base diffusion model weights and finetunes the added ControlNet modules using dual conditioning signals. During inference, a ControlNet scale schedule strategy is introduced to balance controllability and realism. The scale is linearly annealed during the first Ndecay steps: "s(t) = 1 - t/N decay (1 - s min) if t <= N decay, otherwise s min. This prevents over-constraining the generation, leading to smooth, realistic gaits" compared to a fixed scale.

Novel Applications and Results

Tri-Prompting supports novel workflows beyond standard generation. It enables: (1) 3D-aware subject insertion into any scenes and manipulation of existing subjects in an image. For insertion, a harmonized initial frame is created by inserting the character’s initial 2D projection using an image editing model, followed by independent or joint control over camera and object motion. (2) Manipulation of 3D Subjects in a Scene, where users control camera pose and subject motion to manipulate the scene with identity-preserved, motion-controlled generation. Experimental results show Tri-Prompting surpasses Phantom across metrics: it achieves better PSNR/LPIPS than DaS for reconstruction and superior multi-view identity preservation and 3D consistency compared to Phantom.

Key Design Advantages

The unified design brings three main advantages: (1) dual-conditioning signals naturally decouple background and foreground motion; (2) the RGB proxy supports large view changes (e.g., 360° rotations), while multiview images recover missing appearance details and maintain 3D consistency; and (3) low-resolution RGB-based subject motion control generalizes across rigid and non-rigid objects and allows natural object-scene interactions. The framework is data-efficient, fine-tuning with only "11k tuples (<7 hours of video) for <5k steps."

The gist

Tri-Prompting achieves unified control over scene, subject, and motion by integrating scene composition, multi-view subject consistency, and disentangled motion control within a single video diffusion model. The framework resolves coarse motion cues into high-fidelity, 3D-consistent subjects through dual conditioning—XYZ coordinates for background and low-resolution RGB proxies for the foreground—significantly outperforming specialized baselines in both motion accuracy and multi-view identity preservation.

How it works

  1. Tri-Prompting integrates scene composition, multi-view subject consistency, and disentangled motion control within a single model.

Improvements for AI systems

As a fastidious researcher, I have analyzed the Tri-Prompting framework. Here are the specific improvements and capabilities of an AI system built upon this architecture:

)Improved AI System Capabilities: Unified, Fine-Grained Video Synthesis and Manipulation

The improved system will be a unified video diffusion model capable of generating, editing, and controlling complex scenes with unprecedented fidelity across three dimensions: scene composition (Where), subject identity (Who), and motion dynamics (How).

Here are the specific improvements based on the paper's innovations:

Unified Tri-Prompting Architecture for Joint Control:

The system will integrate scene, subject, and motion control into a single diffusion model using a dual-conditioning module. This allows for the simultaneous specification of three distinct prompts:

  • Scene Composition (Where): Controlled by first-frame images and XYZ trajectories derived from background tracking points.

  • Multi-View Subject Control (Who): Controlled by up to three reference images defining the 3D identity of a subject, ensuring appearance consistency across all views.

  • Motion Control (How): Controlled by a dual signal: XYZ trajectories for background motion and downsampled RGB grids for foreground subject motion.

Disentangled Motion Control with Dual Cues:

The system will decouple background scene dynamics from subject movement using two distinct control signals, overcoming the limitations of single-signal methods:

  • Background Motion Control: Utilizes XYZ tracking points (similar to DaS) to guide camera pose and general scene motion.

  • Subject Motion Control: Employs a low-resolution downsampled RGB point proxy grid to provide flexible control over subject movement, enabling realistic non-rigid deformations and handling extreme poses (e.g., 360° rotations) more robustly than purely geometric cues.

Enhanced Subject Identity Preservation via Multi-View Fusion:

The system will maintain high-fidelity identity for the subject across arbitrary pose changes by fusing multi-view images into low-resolution grids during training and inference. This prevents structural distortions (e.g., warping, backward orientation) common in single-view methods like Phantom, ensuring 3D consistency even under complex motions.

Advanced Motion Control via ControlNet Scale Scheduling:

To balance controllability and visual realism during inference, the system will implement a dynamic ControlNet scale schedule. This allows users to transition from highly constrained, controllable motion (low scale) to more natural, realistic dynamics (higher scale) over the denoising steps, resulting in smoother gaits and plausible non-rigid interactions.

Novel Workflows for 3D Subject Manipulation:

The system will enable sophisticated interactive applications that go beyond simple generation:

  • Insertion of 3D Subjects into Scenes: Users can select a background scene, define a 3D character, and control both camera pose (via background XYZ points) and object motion (via subject manipulation) to achieve natural foreground-background interactions.

  • Manipulation of Existing Subjects: Given an image with multiple subjects, the system can reconstruct them in 3D from the first frame and manipulate their poses while maintaining identity consistency, enabling realistic in-scene interaction dynamics.

Improved Data Efficiency and Versatility:

The training paradigm utilizes a two-stage process (LoRA pre-training for scene/subject fusion followed by ControlNet finetuning for motion control), allowing the model to learn foundational identity control before focusing on fine-grained motion guidance, leading to better generalization across diverse styles (anime, movie, real-world).

)What the Improved AI System Can Do: Specific Use Cases

The resulting Tri-Prompting AI system can perform the following specific tasks:

Interactive Video Creation for Content Creators:

Users can select a scene (e.g., a Times Square street image), choose a character from a gallery of multi-view reference images, and use a keyboard interface to simultaneously control the camera trajectory (scene motion) and the character's movement (subject motion). The system will generate high-fidelity video where the character interacts naturally with the environment, maintaining perfect visual consistency regardless of viewpoint changes.

3D Asset Integration into Any Scene:

A designer can provide a 3D model of a custom object (e.g., a vehicle or complex robot) and place it into any existing background image (scene). The AI will generate video sequences where the user can independently control the camera movement around the scene and manipulate the 3D object’s translation/rotation within it, ensuring realistic lighting, shadows, and interaction dynamics.

Real-Time Scene/Subject Manipulation:

In an augmented reality or interactive visualization context, a user could select a subject in a captured video frame and apply motion controls (e.g., walking gait adjustment) while the background scene remains fixed or moves according to scene constraints, achieving precise control over non-rigid object dynamics.

High-Fidelity Subject Synthesis for Virtual Worlds:

The system can generate complex character animations with perfect 3D consistency across multiple angles. This is crucial for creating immersive virtual environments where characters must maintain their appearance and physical integrity while performing extreme actions or transitions, something that current single-view models fail to achieve.

Sources

Related papers