Resolving Representation Ambiguity in Feedforward Novel View Synthesis Transformer via Semantic-Spatial Decoupling

Transformer-based models for feedforward novel view synthesis (NVS) often suffer from representational ambiguity when mixing semantic appearance and spatial information in a shared feature space, leading to artifacts like grid-like distortions. This paper proposes decoupling these two information streams into separate semantic and spatial tokens using an Independent-V attention mechanism, which preserves branch purity while facilitating controlled cross-stream interaction.

The gist

The core architectural change is full pipeline decoupling Transformer by Independent-V Attention mechanism that shares query–key routing for coordination while maintaining two separate value streams.

Problem Addressed: Entanglement and Artifacts

Current architectures like GS-LRM and LVSM mix semantic features (RGB) and spatial features (Plücker rays) into a shared latent space, which causes the spatial bias to interfere with appearance representation. This entanglement results in grid-like distortion [27] on features and leads to degraded rendering. The paper identifies that this mixing makes the spatial bias interfere with appearance representation.

The Decoupled Architecture

The proposed design decouples semantic and spatial tokens into separate branches:

1. For input views, RGB patches are tokenized into a semantic branch (I-branch) and Plücker-ray patches are tokenized into a spatial branch (P-branch). The total token dimension remains the same as the original LVSM token.

2. Target tokens are initialized such that the target semantic branch is initialized with zeros, while the target spatial branch is tokenized from target-view Plücker rays.

Independent-V Attention Mechanism

The Transformer block preserves full-token interaction through shared Query/Key (Q/K) routing on the full token, but uses branch-specific Value and Output projections. The process involves:

1. Computing Q, K from the full token and projecting them to branch-specific value streams: WIvI for the I-branch and WPvP for the P-branch.

2. The shared attention map is computed, but then applied to the two branch-specific value streams: o = concatWIo(AvI),WPo(AvP).

3. After attention, each branch is processed by branch-specific layer normalization (LN) and half-dimensional feed-forward networks (FFNs), before being concatenated back into a D-dimensional token.

Auxiliary Designs: Supervision and Modulation

The decoupled representation enables two optional auxiliary designs to further improve performance:

1. Categorized Supervision: This provides branch-specific training signals. For the I-branch, semantic alignment is achieved via iREPA with DINOv3, and for the P-branch, geometric consistency is enforced using a feature cosine similarity loss (Lgeo).

2. Bidirectional Modulation: This improves interaction between branches by allowing cross-stream conditioning. The spatial representation modulates the semantic branch (Spatial → Semantic) and vice versa (Semantic → Spatial), ensuring controlled bidirectional conditioning while preserving the decoupled representation.

Performance and Analysis

Experiments across decoder-only, encoder-decoder, and iLRM architectures validate effectiveness. The base decoupled design introduces virtually zero additional inference latency compared to the entangled baseline. Ablation studies confirm that decoupling itself provides the main gain, while optional supervision and modulation bring additional improvements (e.g., reaching a 1.1 dB PSNR gain with an 8% inference-time increase when using supervision and modulation). The analysis shows that branch-wise LayerNorm preserves separation, and bidirectional modulation allows spatial tokens to condition appearance tokens, yielding more structured representations.

Limitations and Future Directions

The work focuses on semantic-spatial representation decoupling under controlled feedforward NVS settings. Future directions include studying the same principle at larger training scales, higher rendering resolutions, and more diverse scene distributions, combining decoupled tokens with alternative camera parameterizations, and exploring lighter or fully self-contained training signals that preserve specialization with less offline preprocessing. The paper also discusses potential negative impacts such as the misuse of improved view synthesis techniques to create misleading synthetic visual content.

Broader Impacts

The proposed design improves feedforward novel view synthesis by enhancing the representation of semantic appearance and spatial geometry in Transformer-based models, which has potential positive impacts on more efficient 3D content creation, AR/VR applications, robotics simulation, digital twins, and scalable visual scene understanding. However, it also highlights risks related to the misuse of these techniques for creating misleading synthetic visual content or reconstructing private spaces. Responsible data collection and deployment practices are encouraged to mitigate these risks.

Experimental Reproducibility

The paper provides detailed experimental settings including datasets (RealEstate10K, Objaverse), model configurations (LVSM Decoder-Only, Encoder-Decoder), training schedules (AdamW optimizer, learning rates), loss weights, and compute resources used for training.

Based on the provided scientific paper, here are specific improvements that can be made to existing AI systems and what those improved systems can achieve:

) 1. Decoupling Semantic and Spatial Feature Representations (Core Architectural Improvement)

By replacing the unified token approach with separate semantic (RGB) and spatial (Plücker ray) branches, the model explicitly disentangles appearance from geometry in the latent space.

• This prevents spatial bias from corrupting appearance representation, which is a known cause of artifacts like grid-like distortions.

• The system can now handle complex scenes where texture and geometry are highly correlated (e.g., transparent objects or highly reflective surfaces) without forcing a single feature space to compromise fidelity in either modality.

) 2. Enhanced Cross-Stream Contextual Conditioning (Bidirectional Modulation)

The introduction of bidirectional modulation allows the spatial branch to condition the semantic branch, and vice versa, using learned scale and shift parameters without merging the raw channels.

• This enables spatial cues (like view direction or ray geometry) to precisely modulate appearance updates.

• The improved system can synthesize novel views that are geometrically consistent with a target pose while maintaining high visual fidelity for semantic features (e.g., rendering a specific object texture accurately from an unexpected viewpoint).

) 3. Geometry-Aware Self-Supervision (Categorized Supervision)

By applying branch-specific supervision—Semantic Alignment via DINOv3 alignment and Geometric Consistency via DA3 point map correspondence—the model is trained with explicit, modality-specific constraints.

• The semantic stream learns robust visual features aligned with large pre-trained vision models (DINOv3), improving generalizability across diverse scenes.

• The spatial stream learns to enforce geometric consistency by matching 3D point maps, leading to sharper and more coherent structural details in the synthesized views (as seen in Table 1).

) 4. Improved Feature Quality via Independent Query/Key Routing

By using shared Q/K routing but independent V projections, the model maintains the benefits of global attention coordination while allowing each branch to update its features independently.

• This results in cleaner intermediate representations compared to models where all information is mixed in a single projection, leading to better feature progression (Fig. 4).

) 5. Efficient Inference with Minimal Latency Overhead

The decoupled architecture is designed for efficiency; it incurs virtually zero additional inference latency compared to the baseline LVSM model because the Q/K routing remains full-token based, and supervision modules are discarded at inference time.

• The improved system can perform high-quality novel view synthesis in real-time or near real-time applications (e.g., augmented reality, robotics simulation) without a significant performance penalty due to complex auxiliary modules during deployment.

The improved AI system can now:

1. Generate photorealistic and geometrically accurate novel views of 3D scenes with significantly reduced structural artifacts compared to entangled models.

2. Produce highly coherent outputs in challenging environments (e.g., scenes with complex lighting or view angles) by leveraging explicit spatial-semantic guidance.

3. Be trained more effectively using specialized supervision signals derived from powerful vision models (DINOv3) and precise 3D geometric priors (DA3), leading to superior performance on benchmarks like RealEstate10K and Objaverse.

4. Operate efficiently in deployment environments due to its low inference latency, making it viable for interactive applications like AR/VR or robotics visualization.