Neural Voxel Dynamics: Learning Volumetric Feature Advection for 3D Physics in V-JEPA Latent Space

arXiv:2606.26410 · cs.CV · Submitted 2026-06-24 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Neural Voxel Dynamics".

Jane: Neural Voxel Dynamics presents a self-supervised framework for learning implicit 3D physical dynamics directly from video-derived supervisory signals by shifting the predictive bottleneck from 2D image space to a ‘lifted’…

Tom: First, who's behind it and why it matters.

Paper summary: Tom: Well team, so we're looking at the paper "Neural Voxel Dynamics: Learning Volumetric Feature Advection for three dee Physics in V-JEPA Latent Space," and the core idea is shifting where the prediction happens <ref:2606.26410#pg0>. The thesis is that instead of predicting directly in 2D image space, they move to a 'lifted' three dee Volumetric Latent Space <ref:2606.26410#pg0,to a 'lifted' 3D Volumetric Latent Space>.

Jane: That sounds like a big step because it aims to solve that problem of physical inconsistencies we often see in video generation, which usually lacks a solid three dee geometric foundation <ref:2606.26410#pg0>.

Lu: Exactly, Tom; the paper claims they address the lack of three dee geometric grounding in current generative video models by unprojecting semantic features from Video JEPA into a voxelized grid using monocular depth priors thirty-six <ref:2606.26410#pg1,grid using monocular depth priors 36>. It’s about creating that lifted space to learn implicit three dee physics, which is really exciting because it doesn't rely on explicitly plugging in classical simulators for training or inference <ref:2606.26410#pg0>.

Meng: From an engineering standpoint, I'm interested in how they handle the lifting part. If you’re unprojecting semantic features into a voxel grid while trying to keep it efficient, how much computational overhead does that volumetric representation introduce compared to just working with 2D embeddings <ref:2606.26410#pg0>?

Lalam: I think the way they leverage Video JEPA one nine and then project it into a three dee voxel grid is what makes this approach so potent for cultural applications <ref:2606.26410#pg1,into a 3D voxel grid>. Imagine AI systems that can predict complex physical interactions like fluid flow or rigid body motion just by looking at a video.

Tom: That’s the essence of it, Lalam; they’re not just making pretty videos anymore, they're learning how things actually behave in three dimensions through this volumetric advection process.

Jane: So what does this lifting actually enable them to do in terms of simulating different kinds of phenomena?

Lu: It enables the model to learn a transition operator that treats physics as a spatio-temporal state advection problem within that three dee latent volume, which allows it to unify the prediction of heterogeneous materials like fluids, smoke, and rigid bodies seventeen forty <ref:2606.26410#pg2>. This is a lot for an architecture to handle.

Meng: Unifying those materials implicitly is interesting because traditional hybrid methods often require you to manually specify material properties or rely on external engines like PyBullet or MuJoCo five, which introduces that bottleneck of needing precise system states as input <ref:2606.26410#pg2,on external engines like PyBullet>.

Lalam: That's where this paper really shines; it tracks the material states implicitly within the high-dimensional V-JEPA features, meaning you don't have to manually specify every physical interaction for the model to learn it on its own.

Paper summary: Tom: So, to wrap up this summary of "Neural Voxel Dynamics: Learning Volumetric Feature Advection for three dee Physics in V-JEPA Latent Space," the authors claim they can simulate complex interactions like fluid pouring into a glass by operating in a lifted latent volume without needing handcrafted solvers one nine <ref:2606.26410#pg1,by operating in a lifted latent volume>.

Jane: And the main point is that this framework learns these implicit three dee physics directly from video-derived supervisory signals, moving the predictive bottleneck away from just the 2D image space and into this richer three dee representation <ref:2606.26410#pg0,directly from video-derived supervisory signals>.

Lu: It’s a clever way to ground generative models geometrically while keeping them materially implicit, which is a significant move compared to earlier approaches that used 2D physics proxies like PhysGen thirty <ref:2606.26410#pg2,2D physics proxies like PhysGen 30>.

Meng: I'm still thinking about the practical side—if this volumetric feature advection is the core, how robust is it when dealing with very fine details or highly non-deterministic physical events that might fall outside the learned distribution?

Lalam: The paper addresses that by using a Diffusion Transformer variant with both local spatial and temporal attention modes, which lets each voxel attend to its own trajectory across time, trying to unify those complex physical interactions twenty-eight <ref:2606.26410#pg2>.

Tom: That’s what they’re doing, Lalam; they're not just running a simple solver; they are learning the dynamics by conditioning the feature flow on an action representation uT', which encodes things like point of contact and applied force.

Jane: It sounds like the objective is to decouple structural dynamics from geometric occupancy, using three specific loss functions: feature velocity loss Lfeat, spatially-aware occupancy and occlusion losses Locc and Lobs, and a projection loss Lfeatproj <ref:2606.26410#pg0>.

Lu: The authors are focused on making sure the features they predict actually correspond to the scene content by using an occupancy-weighted mask in that feature velocity loss, which is smart because it forces dynamics to be learned only over regions containing scene content <ref:2606.26410#pg0>.

Meng: Focusing on geometric boundaries through voxel-wise focal loss for the occupancy and occlusion losses, Locc and Lobs, seems like a practical way to ensure the model pays attention to where things are physically touching rather than just predicting empty space <ref:2606.26410#pg0>.

Lalam: And they also have that projection loss Lfeatproj which uses a NeRF-like ray-based depth projection, making sure the final projected 2D latent state stays within the valid V-JEPA latent space <ref:2606.26410#pg0>.

Tom: It’s a multi-layered optimization strategy to make sure the learned three dee dynamics are both physically plausible and structurally grounded in the scene, which is important because we need consistency across different frames <ref:2606.26410#pg0>.

Jane: So, to summarize this paper's core contribution, "Neural Voxel Dynamics: Learning Volumetric Feature Advection for three dee Physics in V-JEPA Latent Space," it proposes a self-supervised method that learns implicit three dee physics by shifting the predictive bottleneck to a lifted three dee Volumetric Latent Space <ref:2606.26410#pg0,to a lifted 3D Volumetric Latent Space>.

Paper summary: Lu: It’s essentially taking video semantic features, unprojecting them into a voxel grid using depth priors, and then using Volumetric Feature Advection to learn an action-conditioned transition operator for simulating things like fluid flow or rigid body motion seventeen forty <ref:2606.26410#pg2>.

Meng: The implication here for the practical world is that we could potentially build generative AI systems that understand physical interactions in a way that doesn't require us to pre-program every single physics rule manually into the model architecture.

Lalam: For culture, this means AI could be much better at simulating complex physical environments in training data, leading to more realistic and consistent creative outputs across different modalities.

Tom: It really moves us toward a system where the AI learns the underlying physical laws through observation rather than being explicitly programmed with those laws beforehand.

Jane: And it’s important to remember that the paper does have limitations; for instance, the authors flag that this method doesn't explicitly model underlying physical forces or three dee interactions, which is why they rely on a generative prior instead of a handcrafted solver <ref:2606.26410#pg2>.

Lu: That’s the trade-off: you get implicit dynamics without needing precise material properties as input, but you might lose some fine causal consistency if the learned advection operator doesn't perfectly capture every subtle force interaction.

Meng: So, while it’s powerful for general simulation, I wonder how easy it is to adapt this to highly specialized domains where we need absolute fidelity in contact mechanics or material stress calculation.

Lalam: The future work suggested in the paper seems focused on extending this latent space modeling further, potentially allowing for even richer interaction types by refining the action-conditioned transition operator twenty-eight <ref:2606.26410#pg2>.

Tom: So, for "Neural Voxel Dynamics," the authors are laying down a powerful framework that treats physics as a spatio-temporal state advection problem within a three dee representation derived from V-JEPA features <ref:2606.26410#pg0,that treats physics as a spatio-temporal state advection problem>.

Jane: The conclusion is that by shifting the prediction bottleneck to this lifted three dee latent space, they can learn implicit three dee physics directly from video signals, achieving geometric grounding while maintaining material subtlety <ref:2606.26410#pg0>.

Lu: This approach provides a new pathway for generative models to handle complex physical scenes without being rigidly constrained by external classical simulators like MuJoCo or PyBullet forty-five <ref:2606.26410#pg2>.

Meng: From my view, the impact is that we might see a generation pipeline where the physics simulation isn't a separate, heavy engine but is baked into the generative latent space itself.

Lalam: It could mean AI systems can generate highly realistic physical scenarios with much more fidelity regarding material behavior and object permanence than what’s currently possible.

Conclusion: Tom: So we've been looking at how Neural Voxel Dynamics tackles learning three dee physics using video data, and now we’re getting to wrap up the title and who actually wrote this paper.

Jane: It really is a mouthful, isn't it? "Neural Voxel Dynamics: Learning Volumetric Feature Advection for three dee Physics in V-JEPA Latent Space." It sounds incredibly technical, but at its heart, it’s about taking what we already have—video understanding—and using it to figure out how things move in the real world.

Lu: The authors are smart because they've managed to bridge the gap between high-dimensional semantic features and a concrete three dee grid representation without needing tons of explicit training data for physics simulations.

Meng: From an engineering standpoint, I like that they've focused on feature advection rather than trying to train a whole new classical solver from scratch; that makes it much more tractable for real-world deployment.

Lalam: I think the real power lies in how this system allows us to imbue generative models with a sense of physical consistency, which is a huge step for creating truly believable digital content.

Tom: Exactly, Lalam; it’s about giving our generative tools a deeper understanding of reality by embedding dynamic motion directly into their latent space.

Jane: So when we look at the authors, they’ve done some really clever work in factorizing the problem into geometric lifting and then feature advection, which is a very sophisticated way to organize complex learning tasks.

Lu: Their methodology is quite elegant; by unprojecting features into voxels using monocular depth priors, they synthesize that crucial three dee context needed for accurate spatial reasoning.

Meng: That synthesis part with the MoGe priors is where I'm looking closely; it’s a clever shortcut to get usable geometry without having to build massive 4D datasets for training.

Lalam: And this capability, when applied broadly, could mean we can generate simulations for anything from complex fluid dynamics in science to realistic cloth folding in design.

Tom: That's the potential impact we're talking about; moving beyond just generating pretty pictures to generating physically plausible scenes that behave correctly under gravity and friction.

Jane: It’s a big conceptual shift because it suggests that learning physical laws can be an emergent property of well-structured latent space dynamics rather than something we have to hand-code entirely.

Lu: Their conclusion really hammers home the idea that this self-supervised framework is capable of learning implicit three dee physics directly from observational data streams.

Meng: I wonder what happens when we push this further into domains requiring extreme precision, like high-speed contact mechanics; I’m curious about the robustness there.

Lalam: That's exactly where the future work comes in; they are looking at refining that transition operator to handle even more intricate interaction types.

Tom: So, to sum up this segment, Neural Voxel Dynamics is using video features to learn how three dee objects move by treating physics as a flow problem in a lifted latent space.

Jane: It’s a big deal because it makes the underlying physical behavior of scenes part of the model’s structure itself.

Lu: The next thing we need to look at is how they handle those specific action-conditioned transitions and what that means for controllable physics generation.

University College London · Adobe Research

cs.CV

Submitted: 2026-06-24

Updated: 2026-10-02

Comments: Project page: https://wangzican.github.io/publication/nvd/

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 91/100

The gist: Neural Voxel Dynamics presents a self-supervised framework for learning implicit 3D physical dynamics directly from video-derived supervisory signals by shifting the predictive bottleneck from 2D

Key concepts

V-JEPA
Joint-Embedding Predictive Architecture is used as the starting point for video understanding. It provides rich semantic representations from video that are then unprojected into a 3D voxel grid using depth information, creating a unified spatial-temporal representation of scene content.
Geometric Lifting
This stage transforms 2D image features into a canonical 3D latent voxel grid. It achieves this by leveraging monocular depth priors to synthesize multiple cameras and depth channels, allowing the model to factorize the problem and capture both scene geometry and semantic content simultaneously in a single representation.
Volumetric Feature Advection
This is the core dynamics learning mechanism. It acts as an action-conditioned generative transition operator that updates voxel frames by simulating a flow in 3D space. It predicts the next state based on external force representations, effectively modeling physics through spatio-temporal state advection.
Diffusion Transformer (DiT) Variant
This variant is used within the Advection operator to govern how voxels evolve over time. It alternates between local spatial attention for capturing localized interactions and temporal attention to allow each voxel's trajectory to be considered across time, enabling the model to capture non-deterministic physical interactions.

Terminology

Summary

Neural Voxel Dynamics presents a self-supervised framework for learning implicit 3D physical dynamics directly from video-derived supervisory signals by shifting the predictive bottleneck from 2D image space to a ‘lifted’ 3D Volumetric Latent Space. This method enables the emergent simulation of heterogeneous phenomena, such as rigid body motion in fluid flow, within a single pipeline without relying on explicit classical simulators.

The gist

Our approach leverages the rich semantic representations of Video JEPA (Joint-Embedding Predictive Architecture) [1, 9], which we unproject into a 3D voxel grid using monocular depth priors [36]. This lifting enables a Volumetric Feature Advection to learn an action-conditioned transition operator that treats physics as a spatio-temporal state advection problem in 3D, i.e., learn implicit 3D physics.

Geometric Lifting to Voxel Latents

The framework begins by tackling the challenge of entanglement between camera perspective, scene geometry, and material dynamics by factorizing the problem into two stages: (i) Geometric Lifting, where semantic features are unprojected into a canonical 3D latent voxel grid; and (ii) Implicit Feature Advection. To achieve this lifting without requiring large-scale 4D data, the model leverages monocular depth priors via MoGe [48] to synthesize N cameras and a depth channel from per-view video. The patch-level embedding from the V-JEPA encoder is then aggregated and unprojected into a unified 4D spatial-temporal voxel representation, resulting in a dense voxel tensor, Zt:= (Ht, Ot, Ut), containing both semantic content and geometric support.

Implicit Feature Advection

The core of the dynamics learning is achieved through Volumetric Feature Advection, an action-conditioned generative transition operator that performs grid-to-grid updates. This predictor learns to simulate complex interactions by operating in a lifted latent volume using a flow-matching formulation [28]. The model predicts the subsequent voxel frame ZˆT′+1 conditioned on an external force representation uT′, where uT′ is a 12-dimensional vector encoding the point of contact, applied direction, magnitude, and temporal duration. To govern this evolution, we employ a Diffusion Transformer (DiT) variant that alternates between two interaction modes: (i) Local Spatial Attention, mimicking localized interactions in classical simulators by attending to a k ×k ×k sparse neighborhood; and (ii) Temporal Attention, where tokens are regrouped by spatial location to allow each voxel to attend to its own trajectory across time. This allows the model to unify and capture non-deterministic physical interactions without rigid constraints of a predefined analytical solver.

Training Objectives

The model is optimized via a multi-task objective that decouples structural dynamics from geometric occupancy, decomposing the predicted voxel state into feature channels Ht, an occupancy probability channel Ot, and an occluded probability channel Ut. The primary loss functions include: (i) Feature velocity loss, Lfeat, which uses an occupancy-weighted mask to ensure dynamics are learned exclusively over regions containing scene content: Lfeat:= P(Ot+1(j)vθ(j) − v⋆(j))∥2. (ii) Spatially-Aware occupancy and occlusion loss (3D), Locc and Lobs, which employ a voxel-wise focal loss to down-weight the contribution from easy-to-predict empty voxels, forcing the model to focus on geometric boundaries: Locc:= 1/omega Σ Xj∈omega FL(Oˆt+1(j), Ot+1(j)). (iii) Projection loss (2D), Lfeatproj, which uses a NeRF-like ray-based depth projection to enforce that the projected voxel state remains in valid V-JEPA latent space: Lfeatproj:= P Σ u,v Mu,v [zˆ(i)t+1(u, v) − z(i)t+1(u, v)] · D.

Latent Projection and Evaluation

To bridge the 3D voxel representation back to the 2D latent space for final video rendering, we employ a ray-based depth projection procedure. This involves casting one ray per pixel into the occupancy volume and recording where it first hits occupied space to determine the predicted soft silhouette Mˆ(i)t+1(u, v). The projected 2D latent feature is then synthesized by weighting the predicted 3D latent features Hˆt+1(j) according to these compositing weights: zˆ(i)t+1(u, v) = Σ Xj∈R(i)(u,v) wj Hˆt+1(j). This projected feature map is then used as a conditioning signal for a lightweight mapper gθ that converts it into the latent space of a pretrained VAE, which finally decodes the RGB prediction.

Improvements for AI systems

Based on the provided scientific paper, Neural Voxel Dynamics: Learning Implicit 3D Physics via Volumetric Feature Advection, here are specific improvements for AI systems and what those improved systems can achieve:

  1. Improve the physical grounding of generative video models by integrating implicit 3D dynamics.

  2. Enable high-fidelity, physically consistent simulation of complex, heterogeneous phenomena (e.g., rigid body motion interacting with fluid flow) directly from passive 2D video observation, without requiring explicit classical simulators or pre-specified material properties.

  3. Develop a unified latent space representation that inherently encodes both geometric structure and semantic content by lifting semantic features into a grounded, sparse 3D voxel grid (Neural Voxel).

  4. Allow the AI system to maintain long-term object permanence and structural consistency across frames, even under complex interactions or occlusions, due to the enforced spatial invariants of the 3D grid.

  5. Implement an action-conditioned transition operator (Volumetric Feature Advection) that treats physics as a spatio-temporal state advection problem in 3D latent space rather than a simple sequence-to-sequence problem in 2D image space.

  6. Enable the AI to predict future physical states by learning a generative velocity field that transports noisy source distributions toward physically plausible target manifolds, capturing non-deterministic dynamics like fluid turbulence or contact physics implicitly.

  7. Create a novel inference pipeline that allows for perspective-independent forecasting and cross-view consistency:

  8. Allow the AI to simulate and predict the evolution of a scene from novel viewpoints or across multiple temporal scales without needing to re-run an explicit (neural) dynamics engine for every view change.

  9. Produce photorealistic, physically grounded video generation by utilizing a learned mapping from the predicted 3D latent state back into a VAE-space latent image, which is then decoded by a frozen pretrained VAE decoder.

  10. Integrate controllable physical manipulation capabilities by expanding the action space to support dense, multi-point articulated (robotic) manipulation guided by parameterized force vectors in the latent space.

  11. Enhance scene understanding and reconstruction accuracy through multimodal geometric supervision:

  12. Improve geometric fidelity of 3D reconstructions by using high-quality monocular depth priors (e.g., MoGe) during the lifting stage, ensuring that geometric errors do not propagate into incorrect voxel occupancy or degraded feature alignment.

  13. Improve representation robustness by incorporating explicit occupancy and occlusion channels into the latent state, allowing the AI to distinguish between empty space and unseen/occluded regions based on depth information, which is crucial for accurate 3D reasoning.

Abstract

We present Neural Voxel Dynamics, a self-supervised framework for learning 3D latent dynamics from monocular video. While generative video models now produce visually compelling motion, their predominantly 2D representations provide limited geometric structure for modeling and controlling physical interactions. Instead, we learn dynamics in a lifted volumetric latent space. Specifically, we unproject semantic Video Joint-Embedding Predictive Architecture (V-JEPA) features into a voxel grid using monocular depth priors, producing a geometrically grounded representation that retains rich video features. We then introduce Volumetric Feature Advection, an action-conditioned transition model that predicts the evolution of these latent features directly in 3D. Unlike hybrid approaches that assume access to privileged physics-engine states or explicit simulation during training and/or inference, we learn only using video-derived supervision and, if available, actions, allowing a single latent dynamics model to represent heterogeneous material-dependent phenomena, unifying rigid-body and fluid dynamics. We evaluate across multiple datasets (synthetic and real) and unseen scenarios (e.g., new boundary conditions, OOD generalization), measuring both predictive fidelity and 3D geometric consistency. We report physical plausibility with Physics-IQ using videos decoded from predicted latent states and videos generated from predicted optical flow. Our results show that geometrically lifting pretrained video latent representations provides a scalable route toward dynamic world models that learn structured 3D physical dynamics without access to privileged simulator state.

Sources

Related papers