Look-Before-Move: Narrative-Grounded World Visual Attention in Dynamic 3D Story Worlds

As embodied AI and world models increasingly operate in dynamic 3D environments, visual perception must move beyond passively interpreting given observations toward actively deciding what to observe.

The gist

Look-Before-Move proposes a narrative-grounded camera planning framework that decomposes camera planning into observation specification, viewpoint search, and trajectory grounding to organize visual attention before generating motion in dynamic 3D story worlds.

Problem Statement and Motivation

The core problem addressed is how an agent in an executable 3D world must actively determine what to observe before planning how to move, moving beyond conventional camera control which often assumes the observation objective is already specified. This capability, defined as Narrative-Grounded World Visual Attention, requires solving three coupled problems: first, converting high-level directorial intent into visual requirements; second, searching for viewpoints that satisfy both narrative relevance and physical feasibility (avoiding occlusion or collision); and third, grounding selected viewpoints into continuous camera motion that tracks dynamic actors. Existing methods often fail to address how an intent should be inferred from narrative intent and grounded in a physically executable 3D world.

Look Stage: Observation Specification

The first step involves converting directorial intent into executable visual constraints via a Semantic Observation Contract. This contract specifies the observable target subject, semantic relations, visibility conditions, composition preferences, and action cues by utilizing a Perception Agent that samples and renders multi-view preview images to obtain environmental feedback. Subsequently, the Planning Agent uses semantic parsing to convert the abstract intent into a concrete observation contract:

Oi = a∗, Cspatial, Taction = Φ(I, si)

This structured contract narrows the state space for subsequent viewpoint search by specifying what to look at and how to look at it.

Monte Carlo Viewpoint Search

To find the optimal visual landing point, Look-Before-Move employs a Monte Carlo Viewpoint Search driven by a Camera Reflection mechanism. The system drives multiple Camera Agents to generate a large number of candidate shot boards in the 3D space. These candidates are then refined through Tournament-based Reranking using an objective function:

v∗init = arg max v∈V (λ1Vis(v, a∗) + λ2Comp(v, Cspatial) − λ3Occ(v, W)

where Vis measures subject visibility, Comp measures composition rationality, and Occ calculates physical occlusion. Following this competitive selection, the Reflection Agent performs localized camera-pose adjustment by re-rendering the scene to check if constraints are satisfied before obtaining the final viewpoint v∗i.

Move Stage: Temporal Motion Coordination

The second stage involves Semantic Trajectory Grounding, which connects selected viewpoints into continuous, collision-aware motion. This is divided into three sub-processes:

1. Trajectory Planning: The Planning Agent uses discrete shots as key nodes to solve for the optimal continuous trajectory τ∗i in the joint spatial-temporal space by minimizing deviation from key shots while constraining kinematic properties like velocity and acceleration.

2. Trajectory Reflection: The Evaluation Agent intervenes to trigger Trajectory Reflection, simulating execution and scoring the generated trajectory τi using a comprehensive formula R(τi) = αPsubj(τi) + βIintent(τi, I) + δQtraj(τi). Only segments exceeding a threshold are retained.

3. Temporal Semantic Editing: The Editing Agent schedules observation clips by maximizing information transmission utility U and minimizing visual incoherence penalty Dtrans to generate the final observation sequence E∗.

Benchmark and Evaluation

To rigorously test this capability, the framework is evaluated on a dynamic 3D Story World Benchmark built using StoryBlender, which covers 50 stories, 457 scenes, and 1585 shots with animated characters and executable environments. Evaluation is conducted across three dimensions: Subject Perception (SP), Intent Consistency (IC), and Trajectory Quality (TQ). The final performance is reported using a segment-count weighted protocol to ensure that the method balances clip quality against switching smoothness, demonstrating that Look-Before-Move improves over representative baselines across all three evaluation dimensions.

Main Results

Quantitative results show that Look-Before-Move achieves the best overall performance with an Overall score of 71.70, outperforming the strongest complete baseline CCD by 6.08 points, with clear gains in both intent consistency and trajectory quality. Qualitative analysis confirms that the method identifies intended characters, renders them at close-up scale, and preserves action cues while selecting physically valid and narratively specific views. Ablation studies confirm that viewpoint search (MLS) and temporal grounding (TG) are crucial for improvement; removing MLS causes a large drop in score, confirming the importance of candidate expansion before motion planning.

Conclusion

Look-Before-Move successfully formulates Narrative-Grounded World Visual Attention as a camera planning capability by decoupling spatial viewpoint composition and temporal motion coordination.

As a fastidious and diligent researcher, I have analyzed the provided paper, Look-Before-Move: Narrative-Grounded World Visual Attention, and distilled actionable improvements for AI systems.

Here are the specific improvements derived from this framework, categorized by the capability they enable:

) Improvements to AI Systems Based on Look-Before-Move Framework:

1. (From Semantic Observation Contract & Planning Agent) Implement a mechanism that converts abstract, high-level directorial intent (e.g., The assassin points a pistol) directly into a structured, executable mathematical constraint set that explicitly defines required visual evidence (target subject identity, spatial relation vector, required action cue preservation).

2. (From Monte Carlo Viewpoint Search) Integrate a multi-agent search process where specialized agents perform coarse sampling based on the contract and then use competitive reranking (Tournament-based Reranking) against a loss function that simultaneously maximizes narrative relevance and minimizes geometric feasibility (occlusion/collision risk).

3. (From Camera Reflection Mechanism) Introduce a closed-loop, render-evaluate-adjust cycle where the system iteratively refines candidate viewpoints by measuring specific sub-metrics: subject visibility robustness, semantic target fidelity (e.g., ensuring the hands are visible if requested), and composition adherence, rejecting any viewpoint that fails these constraints.

4. (From Trajectory Planning & Evaluation Agent) Develop a planning module that solves for a continuous trajectory by minimizing a cost function that balances tracking the selected viewpoints with kinematic smoothness constraints (velocity/acceleration limits). Crucially, implement an evaluation agent that performs real-time simulation of the planned trajectory against physical dynamics to correct for predicted collisions or instability.

5. (From Temporal Semantic Editing) Create a shot scheduling module that utilizes information utility metrics (U(ci)) and transition coherence penalties to intelligently splice discrete observation clips. This allows the system to prioritize clips that maximize narrative information density while minimizing visual jarring transitions between shots focusing on different subjects or semantic targets.

) What the Improved AI System Can Do:

The Look-Before-Move framework transforms an AI agent from a passive observer into an active, narrative-aware cinematographer capable of sophisticated, goal-directed visual storytelling in dynamic 3D environments. Specifically, the improved system can:

1. (Active Observation Decison) Instead of generating motion first and then seeing what it gets, the system will intelligently decide that it needs to capture a specific piece of evidence (e.g., I must see the actor's face to show their reaction) before committing any movement, ensuring all subsequent motion is purposeful.

2. (Intent-Driven Framing) The system can translate complex cinematic instructions into precise 3D viewpoint coordinates and camera parameters, ensuring that every frame adheres not just to geometric rules (no collisions) but also to narrative semantics (e.g., always framing the antagonist in a low-angle shot for power).

3. (Robust Subject Tracking Under Dynamic Conditions) The system will maintain high fidelity on target subjects even when they move behind objects or change positions, by continuously re-evaluating the observation contract and selecting a viewpoint that maximizes subject visibility and identity consistency across sequential shots.

4. (Seamless, Story-Coherent Motion) It can generate camera movements that are not just smooth in motion but are strategically paced to reveal narrative beats sequentially, avoiding unnecessary movement while ensuring critical visual information is acquired at the precise moment it is needed for the story progression.

5. (Autonomous Visual Evidence Organization) For long-horizon tasks (like generating a full movie or complex sequence), the system can autonomously plan a series of observations, selecting and scheduling the most informative shots to convey entire plot points coherently, effectively acting as an intelligent visual editor guided by directorial intent.