RoGe: Novel View Synthesis via End-to-End Implicit Reconstruction and Generation

arXiv:2609.02847 · cs.CV · Submitted 2026-09-02 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "RoGe: Novel View Synthesis via End-to-End Implicit Reconstruction and Generation".

Tom: RoGe presents an end-to-end unified framework for novel view synthesis (NVS) that removes the explicit bridge between reconstruction and generation,

Jane: First, who's behind it and why it matters.

Title and authors: Tom: So we’re talking about the paper "RoGe: Novel View Synthesis via End-to-End Implicit Reconstruction and Generation," and it's focusing on how to synthesize novel views using only a few posed images and a camera trajectory. The authors are Xiaolei Lang, Ze Kang, and Zehao Huang, from Xiaomi EV and Northeastern University.

Jane: It’s important to understand that the title itself points out the method: "Novel View Synthesis" achieved through "End-to-End Implicit Reconstruction and Generation." Essentially, they’re not just making a reconstruction and then feeding it into a generator; they are doing both simultaneously in one integrated system.

Lu: What I find compelling about the authors is that they are explicitly addressing the limitations of previous hybrid methods, which often suffer from errors inherited from lossy projections or a lack of feedback to correct reconstruction mistakes. They want to eliminate that decoupling entirely.

Meng: That direct information flow between the models during training sounds promising for stability, but I wonder how they handle the initial phase where they have no scene understanding at all—how does the feed-forward reconstruction model even start building that implicit representation from just sparse views?

Lalam: The concept of building an implicit scene representation that is directly renderable or queryable from arbitrary camera rays is very powerful; it means the AI develops a deep, inherent understanding of the scene's structure, not just a set of pixels.

The paper's summary: Tom: Moving on to what they actually did in this paper, RoGe proposes a framework where they build an implicit scene representation from sparse inputs using a feed-forward reconstruction model. Then, they query that implicit scene with target camera rays to pull out geometric cues that condition a video diffusion model.

Jane: That’s the core mechanism: first you get the geometry from the reconstruction part, then you use those specific geometric features to guide the generation part. They train these two modules together so that when the generator makes a decision, it’s already being guided by accurate scene geometry derived from those rays.

Lu: The key takeaway here is that they aren't using explicit three dee intermediate representations like depth maps or point clouds; instead, they rely on tokens from the feed-forward reconstruction model forming an implicit representation that can be queried on the fly. That’s a significant simplification for implementation.

Meng: So, if I understand this correctly, it’s not about creating a full three dee mesh first and then projecting it; it’s about extracting features directly from the reconstruction tokens to condition the video diffusion model during training. That makes sense for practical deployment speed.

Lalam: This direct injection of geometric cues into the video generation process is what makes this paper so impactful because it allows the generative objective to shape its own learned geometry, which is a much more holistic learning process.

The paper's improvements: Tom: When we look at how RoGe improves upon prior work, the authors highlight that their method conditions the generation part on per-view geometric features obtained by querying the implicit scene representation with camera rays, which they claim is more effective than using raw reconstruction tokens or decoded RGB maps as conditioning.

Jane: That’s a major point because it suggests that we don't need to feed the generator raw reconstructed pixels; instead, giving it specific geometric information derived from ray queries provides a much stronger signal for consistency.

Lu: The authors demonstrate that joint training brings further gains because the generation objective actively guides the learning of these geometric representations during optimization. This feedback loop is what they argue makes their system better than methods where reconstruction and generation are decoupled.

Meng: If the generative model can directly shape its conditioning, it means we don't have to spend as much time hand-tuning how much reconstruction versus generation we want to prioritize; the AI figures out the optimal balance itself through that shared loss function.

Lalam: This capability speaks to a future where complex AI systems can self-correct their internal representations based on the desired output quality, which is a step toward truly autonomous visual understanding.

Conclusion: Tom: So, to wrap up the discussion on "RoGe: Novel View Synthesis via End-to-End Implicit Reconstruction and Generation," we’ve seen how they unify reconstruction and generation by using ray-queried geometric features as the conditioning signal for a video diffusion model. It seems this end-to-end approach solves the problem of having one part inherit errors from another.

Jane: Exactly, Tom; the paper shows that when you train the models jointly, allowing the generative objective to shape its own geometric conditioning, you get superior results on both image and video metrics compared to existing reconstruction-based or purely generation-based techniques.

Lu: The implication here is that we can move toward systems where scene understanding is learned through a continuous interaction between geometric observation and visual creation, bypassing the need for explicit three dee models in many cases.

Meng: From a practical standpoint, this suggests that for applications requiring high fidelity and geometric consistency from sparse data, this integrated approach might offer more robust results than pipelines that rely on sequential steps with separate error correction mechanisms.

Lalam: I think the most profound implication is how this architecture could improve culture by showing that deep visual understanding doesn't need to be broken down into isolated modules; it can emerge from a tightly coupled system where reconstruction and generation mutually inform each other.

Tom: Fantastic points, everyone. We’ve covered the core mechanism, the joint training benefit, and the specific improvements RoGe offers in terms of geometric consistency from sparse inputs. That was a lot to process!

Jane: Indeed it was; we really got a clear picture of how they're using those ray queries to condition the diffusion model. It’s exciting stuff for anyone working on novel view synthesis today.

Lu: I think the future direction involves exploring how this implicit scene representation can be used for complex tasks beyond just video, perhaps in real-time scene editing or interactive environments where geometry needs constant refinement.

Meng: I wonder if we could see this framework applied to robotics, where the robot needs to synthesize novel views of a room it's only partially sensing during navigation. That’s a very tangible area for this research to impact.

Lalam: It certainly opens up possibilities for creating highly realistic and controllable virtual environments that feel truly grounded in physical reality, which is a massive leap forward for AI-driven content creation.

Xiaomi EV · Northeastern University

cs.CV

Submitted: 2026-09-02

Updated: 2026-09-30

Project page: https://jerry-locker.github.io/roge/ABSTRACT

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: RoGe presents an end-to-end unified framework for novel view synthesis (NVS) that removes the explicit bridge between reconstruction and generation, allowing a generative model to directly consume

Key concepts

Implicit Scene Representation
This is a scene model that doesn't rely on explicit 3D data like depth maps or point clouds. Instead, it uses a network (Camera-Conditioned VGGT) to create tokens that capture the scene's appearance and structure in a way that can be queried directly by arbitrary camera rays, making it flexible for synthesis.
Geometric Feature Extraction
This process involves querying the implicit scene representation with specific camera rays. Two methods are used: a tokenized ray map to query the implicit tokens and a packed ray map supplied directly to the diffusion transformer. This interaction extracts crucial geometric cues that condition the video generation, proving more effective than using raw reconstruction tokens.
End-to-End Joint Training
The system is trained by combining two losses: a Flow Matching Loss (LFM) for generating target velocities and a Rendering Loss ($ ext{renderLrender}$) to supervise the implicit geometry against ground truth images. This joint optimization forces the generation objective to directly shape the geometric conditioning, leading to better visual and geometric consistency.
Latent-Aligned Geometry Adapter
This component bridges the gap between reconstruction features (Fr) and VAE latents (X0). It projects these reconstruction features into a latent space and packs them into an 'implicit geometry condition' (G). This condition, along with appearance and ray information, forms the complete input set for the video diffusion transformer.

Terminology

Summary

RoGe presents an end-to-end unified framework for novel view synthesis (NVS) that removes the explicit bridge between reconstruction and generation, allowing a generative model to directly consume geometric conditioning derived from an implicit scene representation. This approach is significant because existing hybrid methods often suffer from errors inherited from lossy projections or lack a signal to correct reconstruction mistakes. RoGe targets roaming within a scene anchored by sparse views, synthesizing temporally coherent videos along a user-specified camera trajectory using only sparse observations, thereby achieving superior visual quality, geometric consistency, and camera controllability compared to existing reconstruction-based, generation-based, and hybrid baselines.

Core Framework and Goal

The primary goal of RoGe is to synthesize a temporally coherent video along that trajectory given a sparse set of posed observations and a user-specified camera trajectory. The framework achieves this by building an implicit scene representation from sparse inputs using a feed-forward reconstruction model, querying it with target camera rays to obtain per-view geometric features, and injecting these features into a video diffusion model as conditioning. Crucially, the two modules are trained jointly, enabling the generation objective to directly shape its own geometric conditioning.

Implicit Scene Reconstruction

The reconstruction module builds upon feed-forward NVS techniques by leveraging a pretrained network called Camera-Conditioned VGGT. This network aggregates multi-view information through alternating local and global attention, projecting camera intrinsics and extrinsics into a tokenized representation. The resulting tokens, denoted as scene tokens, form an implicit, appearance-preserving scene representation that remains directly renderable or queryable from arbitrary camera rays, without needing explicit 3D intermediate representations like depth maps or point clouds.

Geometric Feature Extraction

To condition the video generation model, RoGe queries this implicit scene representation with per-view rays. This involves two complementary encodings for the Plucker ray map: a tokenized ray map for querying the implicit scene and a lossless packed ray map directly supplied to the diffusion transformer. The interaction occurs via Ray–Scene Cross-Attention blocks, where the ray tokens query the scene tokens to extract geometric cues. These extracted features are then reshaped into spatial feature maps, which can be used to supervise reconstruction during joint training.

Reconstruction-Generation Fusion

The fusion of the two modules is achieved through a hybrid latent representation and a geometry adapter. The sparse context images and target video are encoded using a hybrid latent representation, where context images are encoded independently as single-frame videos, while the target video is jointly encoded from ground-truth frames. A Latent-Aligned Geometry Adapter bridges the gap between the reconstruction features (Fr) and the VAE latents (X0), projecting Fr into a latent space and packing it to form an implicit geometry condition (G). This geometry condition, along with appearance conditions (Y) and ray information (P), forms the complete conditioning set C for the diffusion transformer.

End-to-End Joint Training

The entire system is optimized in a single step using a combined loss function: L = LFM + λrenderLrender. The Flow Matching Loss (LFM) is applied only to the target latent slots, training the model to regress the target velocity based on the conditioned diffusion model output. The Rendering Loss (λrenderLrender) anchors the implicit representation by supervising its decoding into RGB maps against ground-truth images from sparse observations. This joint training allows the generation objective directly shapes its own geometric conditioning, leading to superior performance across image-level and video-level metrics.

Key Contributions

The main contributions include:

  1. Proposing RoGe, a framework that connects a feedforward reconstruction network with a video generation model in an end-to-end manner.

  2. Conditioning the generation part on per-view geometric features obtained by querying the implicit scene representation with camera rays, which are shown to be more effective than raw reconstruction tokens and decoded RGB maps.

  3. Demonstrating that joint training brings further gains, as the generation objective guides the learning of geometric representations.

  4. Achieving state-of-the-art performance on DL3DV by surpassing existing reconstruction-based, generation-based, and hybrid methods on metrics such as PSNR, SSIM, LPIPS (down), and DreamSim (down).

Experimental Validation

Experiments on DL3DV show that RoGe surpasses existing reconstruction-based, generation-based, and hybrid methods across both image-level and video-level metrics. Ablation studies confirm that ray-queried implicit features outperform raw reconstruction tokens. Furthermore, training the modules jointly is shown to further improve all metrics by allowing the generation objective to shape the geometric representation it is conditioned on. The method synthesizes videos with "high visual quality, strong geometric consistency, and precise camera controllability.

Improvements for AI systems

Here are specific improvements to existing AI systems based on the RoGe framework, along with what these improved systems can achieve:

  1. Improved Novel View Synthesis (NVS) for Sparse Inputs: The system can generate temporally coherent, geometrically consistent videos from only a few posed images and a camera trajectory. This overcomes the limitations of reconstruction-based methods (blurring/holes in unobserved regions) and generation-based methods (lack of geometric consistency).

  2. End-to-End Joint Reconstruction and Generation: The system integrates scene reconstruction (via a feed-forward network) directly with video generation (via a diffusion model) without an explicit, lossy 3D intermediate representation. This allows the generative prior to directly guide the learning of the necessary geometric features.

  3. Geometry-Conditioned Video Generation: The improved system conditions its video generation process on per-view geometric features extracted by querying an implicit scene representation with camera rays (Plücker ray maps). This ensures that the generated novel views adhere strictly to the geometry observed in the sparse input views, leading to superior visual quality and geometric consistency.

  4. Enhanced Camera Controllability: The system can synthesize videos along arbitrary, user-specified camera trajectories while maintaining strong geometric fidelity and precise camera control. Ablation studies show that ray-queried features provide a significant gain over raw reconstruction tokens or rendered RGB maps as conditioning, directly improving the accuracy of the synthesized view relative to the target pose.

  5. Improved Extrapolation Capabilities: The system can synthesize novel views along trajectories independent of the original observed poses (extrapolation), leading to geometrically consistent videos even in unseen regions, surpassing methods that suffer from geometric distortions when moving outside observed bounds.

  6. Robustness to Sparse Data: By leveraging a feed-forward reconstruction model and conditioning on ray queries rather than relying solely on dense 3D explicit representations (like full point clouds or meshes), the system maintains high performance even when scene coverage is minimal, making it highly effective in practical, sparse-view scenarios.

Sources

Related papers