Rethinking Object-Centric Representations for Video Dynamics Modeling

arXiv:2606.23436 · cs.CV, cs.AI, cs.LG · Submitted 2026-06-22 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Rethinking Object-Centric Representations for Video Dynamics Modeling".

Jane: Unsupervised video object tracking aims to decompose dynamic scenes into persistent, object-centric entities without manual annotations, and this work proposes STAITUS,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: To wrap up this discussion on "Rethinking Object-Centric Representations for Video Dynamics Modeling," the authors, Amaury Wei, Ismail Nejjar, and Olga Fink, introduced STAITUS. They proposed this unified slot-based video framework specifically to address the failures of existing methods where object appearance and pose get entangled.

Jane: In simple terms, they're suggesting that by explicitly separating the geometry of an object from what it looks like, you can create representations that are much clearer for tracking. The implication is that we can build systems where objects maintain their identity even when they move around or change viewpoint.

Lu: The title itself, "Rethinking Object-Centric Representations," suggests a fundamental shift in how we think about scene decomposition, moving away from the limitations of fixed slot representations to something more dynamic and disentangled.

Meng: I see that STAITUS tackles the problem by using several regularization terms, including a spatial separation loss and a temporal alignment loss, which are designed to keep the visual embeddings clean while respecting the motion constraints. That structured approach is what makes me think it’s viable for implementation.

Lalam: I think the real impact is that by making these representations more robust to motion, we move toward AI systems that exhibit much greater persistence and reliability in tracking dynamic scenes. This has implications for how we design future video understanding tools across many domains.

Tom: So, the main conclusion here is that STAITUS offers a path forward by disentangling appearance and pose to achieve sharper object masks and significantly more stable identities. It's a solid framework for tackling the challenges in unsupervised object tracking.

Jane: And it really highlights how carefully designing the loss functions, like enforcing temporal alignment exclusively in the appearance space, can guide the model toward more meaningful representations. It’s about guiding the learning process intelligently rather than just throwing more data at it.

Lu: The way they handle adaptive slot usage also shows a deep understanding of scene complexity, dynamically deactivating redundant slots to prevent over-segmentation. That level of dynamic control is what makes this approach feel powerful for complex environments.

Meng: I'm glad we touched on the practical side, because the paper does mention that their method relies on features extracted from a pretrained and frozen self-supervised DINO encoder. That suggests it has some good starting points for training without needing massive amounts of labeled data right away.

Lalam: For me, the cultural implication is that this work suggests we can build AI tools that are more trustworthy in situations where object tracking is crucial, making autonomous systems safer and more predictable for human interaction.

Conclusion: Tom: So, to wrap up our discussion on "Rethinking Object-Centric Representations for Video Dynamics Modeling," we've seen how this STAITUS framework tackles that old problem of confusing object appearance with its movement.

Jane: It really boils down to taking those complex video scenes and breaking them down into cleaner, more organized object components using a new way of looking at features.

Lu: I find the idea of disentangling geometry from appearance fascinating; it opens up so many possibilities for how we model interactions between objects in dynamic environments.

Meng: From an engineering standpoint, the core idea is that by separating those two elements, we can build systems that are much more predictable when things are moving quickly.

Lalam: This work suggests a step toward AI tools that can maintain object identities across long sequences, which could really make visual recognition in real-time applications much more reliable for everyday use.

Tom: Exactly, and the authors’ approach with their recurrent attention module and adaptive slots shows they’re thinking about how to handle scene complexity dynamically.

Jane: And when you look at the title, "Rethinking Object-Centric Representations," it really signals a fundamental shift in how we structure our understanding of video content.

Lu: It’s not just a small tweak; it suggests that our current methods for video modeling might be fundamentally missing some key structural information about how objects exist in motion.

Meng: I'm curious about the practical side—how does this improved representation translate into a faster, more efficient tracking system in a production environment?

Lalam: It points toward a future where AI can build richer, more stable cultural artifacts from video data, maybe even helping us understand how objects change context over time.

Tom: That’s what I was thinking—this paper sets the stage for much smarter object tracking systems that don't just follow pixels but actually understand the underlying structure of what's happening.

Jane: It really highlights how carefully designing those loss functions to enforce temporal consistency in appearance space can guide the learning process toward more meaningful results.

Lu: The way they’ve managed to keep the geometric attributes evolving freely while constraining the temporal dynamics in a specific subspace is quite sophisticated design work.

Meng: I do wonder about the scalability; if we move this type of disentanglement into much larger, high-resolution videos, how robust will these slot activations remain?

Lalam: The implication for culture is that as AI gets better at tracking and understanding objects in video, we could see richer interactive experiences where digital entities feel more persistent and less like fleeting moments.

Tom: So the authors have laid a very solid foundation here by showing how to make object representations sharper and identities more stable under motion.

École Polytechnique Fédérale de Lausanne

cs.CV, cs.AI, cs.LG

Submitted: 2026-06-22

Updated: 2026-09-28

Comments: 28 pages, 14 figures

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 90/100

The gist: Unsupervised video object tracking aims to decompose dynamic scenes into persistent, object-centric entities without manual annotations, and this work proposes STAITUS, a unified framework that

Key concepts

Recurrent Adaptive Disentangled Slot Attention Module
This core component iteratively refines object representations across time steps using attention. It replaces fixed positional encodings with slot-specific reference frames derived from object geometry, creating appearance embeddings that are inherently invariant to the object's motion.
Adaptive Slot Activation
A mechanism that uses a lightweight MLP to predict which objects (slots) are active at any given moment. This allows the model to dynamically adjust the number of active slots based on scene complexity, preventing over-segmentation and managing scene dynamics effectively.
Temporal Alignment Loss
This loss function enforces consistency in how an object looks across consecutive frames by measuring cosine similarity between its appearance embeddings. It operates only in the disentangled appearance space, regularizing the visual features of a single object to ensure temporal stability under motion.

Terminology

Summary

Unsupervised video object tracking aims to decompose dynamic scenes into persistent, object-centric entities without manual annotations, and this work proposes STAITUS, a unified framework that explicitly disentangles object appearance from spatial pose to yield sharper masks and more persistent identities under motion.

How it works

  1. The framework begins by extracting dense features from an input video frame using a pretrained and frozen self-supervised DINO [5] encoder, resulting in dense patch features, which are then projected into a task-specific embedding space using a lightweight two-layer MultiLayer Perceptron (MLP) Fproj to obtain initial representations.

  2. The core of STAITUS is the Recurrent Adaptive Disentangled Slot Attention Module, which uses an iterative attention-based grouping module Gslot for slot refinement across time steps. This module builds upon Invariant Slot Attention (ISA) [3] by replacing fixed global positional encodings with slot-specific reference frames derived from the slot geometry, which produces an appearance embedding that is invariant to object motion.

  3. To manage scene complexity and prevent over-segmentation, STAITUS incorporates an Adaptive Slot Activation mechanism. A lightweight MLP Ggate predicts a binary activation variable, allowing the model to dynamically determine which slots are active at each timestep, thereby dynamically adapting the number of active slots to match scene complexity.

How it works (Continued)

  1. The system utilizes three complementary objectives during training:

(a) Reconstruction Loss:

The primary unsupervised learning objective is minimizing the pixel-wise Mean Squared Error (MSE) loss, Lrecon = E∥xt − xˆt∥22 over the input frame and its composited reconstruction, which serves as the primary unsupervised learning signal.

(b) Temporal Alignment Loss:

This loss enforces consistency of each slot’s visual appearance across frames using cosine similarity:

The formula is defined as:

Ltime = (sum over t, k of z k t-1 z k t (1 - cos(v k t, v k t))) / (sum over t, k of z k t-1 z k t). This loss regularizes consecutive slot appearance embeddings using cosine similarity and operates exclusively in the disentangled appearance space.

(c) Spatial Separation Loss:

This loss ensures distinct objects are represented by different slots within a frame, penalizing similar appearance embeddings among active slots:

Lsep = (sum over t sum over i != j of z i t z j t max(0, cos(v i t, v j t))) / (sum over t sum over i != j of z i t z j t). This loss discourages similar appearance embeddings among active slots by penalizing positive cosine similarity.

  1. The final frame reconstruction is achieved through Adaptive Spatial Broadcast Decoding, which maps each slot representation to a full-frame RGB reconstruction xˆk t and an alpha mask α k t using the disentangled geometric parameters (p k t, s k t) to construct a slotspecific coordinate grid. The final frame is then obtained by compositing all active slots:

xˆt = sum over k of z k t · α k t ⊙ x k t.

  1. The full training objective combines these components:

L = Lrecon + λtime Ltime + λsep Lsep + λspars Lspars, where Lspars encourages the model to activate only the required number of slots at each timestep. This structure allows geometric attributes to evolve freely while temporal constraints operate in a semantically meaningful subspace.

  1. The method's effectiveness is validated through extensive experiments on synthetic benchmarks like CLEVRER and MOVi, as well as real-world datasets like YouTube-VIS, demonstrating that STAITUS consistently outperforms state-of-the-art baselines in both segmentation quality and tracking stability. Ablation studies confirm that each regularization term plays a complementary role: temporal alignment is critical under motion, while separation and sparsity are most important in visually complex scenes.

  2. STAITUS achieves superior results by explicitly addressing the failure mode where existing slot-based models entangle object appearance with pose, leading to background-foreground mixing and identity switching, thus producing sharper object masks and more stable identities. The reliance on DINO features and the structure of the disentangled representation are shown to be robust across different pretrained visual backbones.

Improvements for AI systems

Here are the specific improvements for AI systems based on the STAITUS framework, and what those improved systems can achieve:

The STAITUS framework introduces a unified slot-based video representation that explicitly disentangles object appearance from geometric pose (position/scale), enabling temporally aligned appearance modeling, spatial separation between objects, and adaptive slot usage.

Here are the specific improvements and capabilities of an AI system built upon STAITUS:

  1. The system can perform unsupervised object tracking in videos with significantly higher stability than prior methods (like SAVi or VideoSAUR).

  2. The system produces sharper object masks with minimal background leakage, leading to superior foreground-background separation compared to baselines (e.g., DINOSAUR, SlotContrast).

  3. The system maintains consistent object identities over long video sequences, successfully avoiding identity swapping and object grouping errors seen in previous models (like SlotContrast or VideoSAUR).

  4. The system can dynamically adapt the number of active slots based on scene complexity, preventing over-segmentation by deactivating redundant background slots.

  5. The system is robust to changes in camera motion, object occlusion, and viewpoint variations because its temporal alignment loss operates exclusively in the disentangled appearance space, allowing pose attributes to evolve freely under motion.

These improved AI systems can achieve the following specific tasks:

  1. A high-fidelity video segmentation pipeline that outputs precise masks for every detected object across an entire sequence without manual labeling.

  2. Robust tracking of moving objects in complex, real-world scenarios (e.g., autonomous driving or surveillance) where objects may enter, exit, or be occluded by other dynamic elements.

  3. Accurate scene decomposition into distinct semantic regions while maintaining sharp boundaries between foreground objects and the background (e.g., separating a person from a forest in a video).

  4. A scalable object discovery system capable of handling scenes with varying numbers of objects and high visual complexity, as it dynamically manages its representation capacity.

Abstract

Learning to decompose videos into persistent objects is a fundamental challenge in unsupervised object-centric representation learning. Despite recent progress, existing methods struggle to simultaneously achieve accurate object segmentation, consistent identities over time, and reliable foreground-background separation. To address these challenges, we introduce UniSlot (Unified Slots), an unsupervised framework for learning robust and disentangled object-centric representations from videos. UniSlot explicitly separates object appearance from its 3D-aware geometric pose in the scene, linking object identity to appearance while leveraging depth to better distinguish objects from their surroundings. UniSlot achieves state-of-the-art performance in unsupervised object-centric video decomposition and tracking across synthetic and real-world benchmarks, yielding substantially tighter object masks and reducing background leakage while preserving object identities. Beyond decomposition and tracking, these improved representations translate directly to downstream tasks such as unsupervised object dynamics prediction, enabling more accurate forecasting of future object trajectories.

Sources

Related papers