CST-WM: A Causally Structured World Model for Embodied Visual Tracking

arXiv:2609.06302 · cs.CV, cs.RO · Submitted 2026-09-05 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "CST-WM: A Causally Structured World Model for Embodied Visual Tracking".

Tom: Embodied visual tracking requires robots to make predictive decisions over future target observability and apparent scale, especially when dealing with occlusion or distractors.

Jane: First, who's behind it and why it matters.

Title and authors: Tom: So, we're talking about CST-WM: A Causally Structured World Model for Embodied Visual Tracking today, and the authors are from New York University Abu Dhabi.

Jane: The title itself really tells you what this research is focused on: building a world model that understands the causal structure of tracking in embodied visual tasks.

Lu: They’re tackling the need for robots to make predictive decisions about future target observability and apparent scale, especially when things get occluded or if there are distracting objects around them.

Meng: From an engineering standpoint, this sounds like they are trying to build a system that doesn't just react to the current frame but actually plans based on what the future looks like for the target.

Lalam: It’s about creating a model where action only affects future observations through robot motion and the resulting change in what we observe, rather than injecting direct control effects into the target evidence branch.

Tom: That distinction is key, Lalam. It sounds like they are moving away from models that just correlate control with observation.

Jane: They propose CST-WM by decomposing the latent state into distinct branches: a target-evidence branch, a robot branch, and an observation branch to enforce these causal constraints.

Lu: By doing this factorization, they ensure that the target-evidence branch is updated without direct action injection while keeping action localized to the robot motion and observation updates.

Meng: That structural alignment between prediction semantics and tracking semantics seems like a very smart way to build a more reliable planning system for robots on the ground.

The paper's summary: Tom: Okay, let's talk about what CST-WM actually does, based on the summary they provide in their work. Essentially, they’ve tackled that causal hallucination problem head-on.

Jane: They decompose the latent state into three distinct branches so that the target evidence branch is updated without direct action injection and action is only allowed to reach future observations through robot motion.

Lu: This design ensures that action is excluded from the direct update of the target-evidence branch, which preserves a representation tied to observability and apparent scale rather than absorbing control information.

Meng: So, they are essentially separating *what we see* from *what we do* in the model's internal state structure, which sounds like a very clean way to handle these kinds of dependencies.

Lalam: They extract a target-evidence representation called Hl, which summarizes observability and apparent scale using tools like GroundingDINO for detection confidence and normalized bounding box area.

Tom: So, the full structured state Sl is then formed by integrating that target evidence with an observation latent from a pre-trained VAE, and the robot state recovered from the action history.

Jane: That full structure makes sense; they aren't just predicting one thing but building a rich context that includes what we're looking at, what we know about our surroundings, and where we are moving.

Lu: This structured approach is what allows them to distinguish between different candidate action sequences when rolling out futures, which is something conventional models fail to do effectively.

The paper's improvements: Tom: Now that we understand the structure, let's look at how CST-WM improves things compared to the methods they’re comparing it against. The authors suggest several key improvements in their approach.

Jane: One major improvement is moving beyond reactive or generic world models to this causally structured model for better long-horizon tracking and recovery in cluttered indoor environments.

Lu: They also aim to enable robust, multi-step target re-acquisition after temporary occlusion or distraction by allowing the model to plan based on the predicted evolution of target evidence instead of relying on myopic frame-to-action mappings.

Meng: From a practical standpoint, this means we can expect significantly higher Re-acquisition Success and reduced Time to Re-acquire because the planning isn't just looking at what happened in one moment.

Lalam: Another improvement is enforcing a structural constraint where action information must flow only through the robot's ego-motion branch into future observations, not directly into the target evidence state.

Tom: That structural enforcement is crucial; it directly leads to better planning-value consistency and reduces direct action leakage, which they quantify with Jacobian leakage statistics showing zero for CST-WM compared to a baseline of zero point one zero two in their work.

Jane: Furthermore, they incorporate an auxiliary distance-aware loss during training, which forces the target-evidence branch to correlate its representation with actual relative distance proxies derived from simulator-only relative distance labels.

Lu: This helps improve generalization across different environments and robot scales because this objective makes the tracking system more robust to variations in things like human appearance or body scale.

Conclusion: Tom: So, wrapping up our discussion on CST-WM, it seems the main conclusion is that for embodied visual tracking, the predictive structure itself needs to align with how target evidence enters planning.

Jane: They’ve shown that by decomposing the latent state into distinct branches and enforcing causal constraints on action flow, they can achieve better following quality and re-acquisition in tasks where traditional models struggle.

Lu: The implication here is that for embodied visual tracking, the predictive structure itself must align with how target evidence enters planning, which opens up new avenues for more reliable AI systems in robotics.

Meng: For practical application, this means we can expect better performance metrics across following distance control and safety because the planning value function balances visibility and distance regulation over the entire prediction horizon.

Lalam: It’s encouraging to see how this structural modeling approach helps prevent causal hallucination, which is a big step for building more reliable and trustworthy visual tracking AI systems in real-world scenarios.

Junyi Hu Shuaihang Yuan Yi Fang

New York University Abu Dhabi

cs.CV, cs.RO

Submitted: 2026-09-05

Updated: 2026-09-29

Project page: https://junyi2005.github.io/cst-wm

Importance score: 92/100

The gist: Embodied visual tracking requires robots to make predictive decisions over future target observability and apparent scale, especially when dealing with occlusion or distractors.

Key concepts

CST-WM
A Causal Structured World Model for Embodied Visual Tracking. It is a model designed for robots to make predictive decisions about future target observability and apparent scale, especially when dealing with occlusions or distractors.
Latent State Decomposition
The proposed method decomposes the latent state into three distinct branches: a target-evidence branch, a robot branch, and an observation branch. This factorization ensures that action only affects future observations through robot motion rather than directly controlling the target evidence.
Causal Hallucination Problem
The paper addresses the problem where models generate false information. CST-WM tackles this by ensuring that the target-evidence branch is updated without direct action injection, preserving a representation tied to observability and scale instead of absorbing control information.

Terminology

Summary

Embodied visual tracking requires robots to make predictive decisions over future target observability and apparent scale, especially when dealing with occlusion or distractors. This paper introduces CST-WM, a causally structured world model designed to address the task-specific causal hallucination that plagues current action-conditioned sequential prediction methods. By decomposing the latent state into distinct branches for target evidence, robot motion, and observation, CST-WM ensures that action influences future observations only through robot motion rather than injecting direct causal effects into the target evidence branch. This structural alignment between prediction and tracking semantics is shown to improve following quality and re-acquisition in embodied visual tracking tasks.

Causal Hallucination as a Problem

The central difficulty in embodied visual tracking is a task-specific form of causal hallucination: "in action-conditioned sequential prediction, a model can exploit the strong correlation between robot control and target-related observations by hallucinating a direct causal effect from the current action to target evidence, rather than allowing action to influence such evidence only through robot motion and the resulting observation change. This shortcut yields plausible futures while giving the wrong semantics for tracking-oriented planning and target re-acquisition. Reactive trackers are myopic because they lack a basis for multi-step recovery, while generic world models fail because they allow a generic actionconditioned transition can write the current action directly into the target-evidence branch. The core issue is that the action pathway in that prediction matches the structure of tracking under ego-motion."

CST-WM Architecture and Factorization

CST-WM decomposes the latent state into three branches: a target-evidence branch, a robot branch, and an observation branch. The transition is explicitly factorized to enforce causal constraints:

  1. The target-evidence branch is updated without direct action injection.

  2. Action remains available only to the robot motion and observation updates.

This design enforces the structural requirement: the target-evidence branch is updated without direct action injection, while action is localized to the robot branch and only reaches future observation through the architecturally permitted route. This ensures that action is excluded only from the direct update of the target-evidence branch, preserving a representation tied to observability and apparent scale rather than absorbing control information.

Information Extraction and State Construction

At each timestamp, CST-WM extracts a target-evidence representation, denoted as Hl, which summarizes observability and apparent scale. This is constructed using:

  1. GroundingDINO to obtain detection confidence (HGDINO) and normalized bounding box area (Harea).

  2. The target-evidence token is defined as Hl = [HGDINO(Ol), Harea(Ol)].

The full structured state Sl is then formed by integrating these components: Sl = [Hl, Zl, xl], where Zl is the observation latent from a pre-trained VAE, and xl is the robot state recovered from the action history.

Training and Planning Framework

The model is trained using diffusion-based transitions (DDPM) on dataset pairs like EVT-Bench and Habitat 3.0. The training objective includes:

  1. A primary denoising loss (Ldiff) over the shared next state Sl+1, utilizing causal masking to enforce the architectural constraint.

  2. An auxiliary distance-aware loss (Ldist), which supervises a prediction head gdist on the target-evidence branch using simulator-only relative distance labels derived from simulator-only relative distance dl+1.

Planning is performed using rollout-based Model Predictive Control (MPC) with the Cross-Entropy Method (CEM). The planner seeks sequences that optimize a value function V(S0, a0:T −1), which balances objectives such as:

-βvisHb visHb area

-αHb area-

-λvalidX−1t=0I(at ∈ A/ valid)

-λsafeX−1t=1I(Sbt ∈ O/ safe)

Experimental Results and Diagnostics

Experiments on EVT-Bench and Habitat 3.0 demonstrate improvements over reactive and world-model baselines in metrics such as following quality, distance-range control, safety, and re-acquisition. Offline diagnostics further confirm the structural benefits:

  1. Better multi-step rollout fidelity (measured by latent prediction error).

  2. Stronger planning-value consistency (Spearman’s ρ and Kendall’s τ against simulator rankings).

  3. Substantially reduced direct action leakage, quantified by Jacobian leakage statistics which show 0.00 for CST-WM compared to baselines like the leaky variant C (0.102).

These results suggest that "for embodied visual tracking, the predictive structure itself must align with how target evidence enters planning.

Improvements for AI systems

Here are specific improvements that can be made to existing AI systems, as derived from the CST-WM paper:

  1. Improve long-horizon tracking and recovery performance in dynamic, cluttered indoor environments by replacing reactive or generic world models with a Causal World Model structure (CST-WM). This system will maintain target observability and apparent scale through a dedicated target-evidence branch that is protected from direct action injection.

  2. Enable robust, multi-step target re-acquisition after temporary occlusion or distraction by allowing the model to plan actions based on the predicted evolution of target evidence, rather than relying on myopic frame-to-action mappings. This results in significantly higher Re-acquisition Success and reduced Time to Re-acquire (TTR).

  3. Enhance planning consistency and reliability by enforcing a structural constraint: action information must flow only through the robot's ego-motion branch into future observations, not directly into the target evidence state. This eliminates causal hallucination, ensuring that imagined futures align with how target evidence is actually generated by physical motion, leading to stronger planning-value consistency (higher Spearman’s ρ and Kendall’s τ).

  4. Improve distance regulation and safety in following tasks by utilizing a compact, observation-derived target-evidence representation that summarizes observability and apparent scale. This allows the system to effectively manage following distance without requiring privileged geometric supervision at test time, leading to better Distance-Range Success (DRS) and safety metrics.

  5. Achieve superior generalization across different environments and robot scales by incorporating a distance-aware loss during training. This auxiliary objective forces the target-evidence branch to correlate its representation with actual relative distance proxies derived from simulators, making the tracking system more robust to variations in human appearance, clothing, and body scale (as validated across cross-humanoid benchmarks).

  6. Develop a planning framework that supports both stable following and recovery within a unified model. This is achieved by using rollout-based Model Predictive Control (MPC) guided by a planning value function that explicitly balances target visibility and distance regulation over the entire prediction horizon, allowing for adaptive decision-making in cluttered scenes.

  7. Increase the fidelity of long-horizon predictions by leveraging diffusion training with causal masking, which prevents spurious correlations from propagating through the model during sequential prediction. This results in lower latent prediction error and higher Target-Visibility AUROC across extended horizons (e.g., up to 40 steps).

Sources

Related papers