GenIA: Generative Reconstruction with Test-Time Input Alignment

summary

The gist

The gist: GenIA introduces an inference-time framework that aligns SAM3D’s generative prior to observations using geometric and photometric constraints, improving pose and appearance consistency

In short

GenIA is a framework that aligns SAM3D's generative 3D model to real-world observations during inference without retraining it. It improves object pose by deriving translation and scale from geometry while keeping the learned rotation prior. Appearance consistency is achieved through visibility-biased attention, cross-observation fusion, and differentiable rendering guidance, leading to better reconstruction across monocular, multi-view, and dynamic inputs.

Key concepts

Test-Time Input Alignment: This is the core idea: taking a pre-trained generative model (like SAM3D) and modifying its inference process so that its output matches specific input data (images or video frames). GenIA achieves this by grounding the model's predictions in observed geometry and photometric data at test time, ensuring the generated 3D asset is consistent with what is actually seen.
Geometry-Derived Pose Estimation: Instead of relying solely on image-based pose estimation, GenIA uses depth cues derived from the input to accurately determine an object's translation (where it is in space) and scale (how big it is). This geometric information helps refine the object's world-space placement while preserving the rotation learned by the foundation model.
Visibility-Biased Attention: This technique directs the model's attention mechanism during appearance prediction. It biases the attention towards image patches that actually observe specific voxels, effectively ignoring occluded or unobserved areas. This ensures that only visible parts of the object influence its predicted appearance and shape details.
Differentiable Rendering Guidance: This involves using a differentiable rendering process during the denoising steps of the generative model. It provides an explicit supervision signal by comparing the rendered output from a predicted 3D model with the actual input image. This guidance helps correct geometric and photometric errors in real-time.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Federated Learning — explained in 17 episodes
Sample Complexity — explained in 17 episodes
Retrieval-Augmented Generation (RAG) — explained in 16 episodes
Interpretability — explained in 15 episodes
Quantum Fisher Information (QFI) — explained in 15 episodes

This episode discusses

The paper

GenIA: Generative Reconstruction with Test-Time Input Alignment · Read on arXiv

Stefano Esposito, Naama Pearl, Polina Karpikova, Samuel Rota Bulò, Lorenzo Porzi, Peter Kontschieder, Andreas Geiger

Tübingen AI Center, University of Tübingen

Reconstructing complete 3D object assets from monocular or sparse multi-view observations remains challenging. Generative 3D foundation models can complete object geometry beyond the observed views, but their predictions may not faithfully reproduce the observed geometry, appearance, or pose. We introduce GenIA, a framework for test-time input-aligned generation that grounds SAM3D's generative prior in geometric and photometric observations without retraining the foundation model. We improve object pose by deriving translation and scale from geometry while retaining the learned rotation prior, and align appearance through visibility-biased attention, cross-observation fusion, and differentiable rendering guidance during denoising. An optional post-denoising refinement further adapts the appearance latent, lightweight decoder adapters, and object placement to the observations. Our framework also supports externally supplied geometry; when given temporal shapes of dynamic objects, it recovers a shared, input-aligned canonical appearance and stable world-space placement. Across synthetic and real benchmarks, GenIA improves pose prediction and object reconstruction from monocular, multi-view, and dynamic inputs, outperforming recent optimization-based, per-frame image-to-3D, and video-to-4D methods. Our project page is available at https://facebookresearch.github.io/GenIA.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "GenIA: Generative Reconstruction with Test-Time Input Alignment".

Tom: The gist: GenIA introduces an inference-time framework that aligns SAM3D’s generative prior to observations using geometric and photometric constraints, improving pose and appearance consistency across monocular, multi-view,

Jane: First, who's behind it and why it matters.

Title and authors: Tom: Let's talk about who wrote this. The paper is titled "GenIA: Generative Reconstruction with Test-Time Input Alignment," and the authors are Stefano Esposito, Naama Pearl, Polina Karpikova, Samuel Rota Bulò, Lorenzo Porzi, Peter Kontschieder, Andreas Geiger and Jonathon Luiten.

Jane: They're a solid team from the Tübingen AI Center at the University of Tübingen. It shows they're working on cutting-edge research in three dee foundation models and generative reconstruction <ref:2610.12388#pg1>.

Lu: What this title really tells us is that they are focusing on alignment during inference, which means making sure the output you get when you run the model matches your input data right when it happens, rather than fixing it during the initial training phase.

Meng: So, if you're driving a car and need a quick three dee understanding of something in front of you—maybe a pedestrian or an obstacle—this work aims to make that process more accurate on the fly <ref:2610.12388#pg1>.

Lalam: It implies they are trying to bridge the gap between having a model that can generate complete three dee shapes and actually being able to use that shape accurately in real-world, input-dependent scenarios <ref:2610.12388#pg1>.

The paper's summary: Tom: So, what's the actual gist of GenIA? They are showing how to align generation to observations at three different levels. First, they recover translation and scale using depth cues while keeping the generative rotation prior.

Jane: Then they condition appearance on geometry and visibility, directing local features toward the image regions that observe them, and fusing predictions across views or frames with visibility weighting.

Lu: And finally, they use differentiable rendering to provide both in-trajectory guidance for dynamic inputs and an optional post-denoising refinement step. That’s a multi-pronged approach to getting the geometry, pose, and appearance right all at once.

Meng: It sounds like they are moving beyond just one image—monocular input—to handle multi-view and dynamic inputs with this framework.

Lalam: The paper explains that their grounding mechanisms are agnostic to the specific geometry predictor being used, meaning it can even support externally supplied shapes if you have them available.

The paper's improvements: Tom: The main improvement is how they handle pose prediction. They use SAMthree dee’s coarse geometry model to predict the world-space pose, but they estimate translation and scale from geometric cues instead of relying solely on image-space errors.

Jane: They measure three things from the camera-space pointmap: the object’s depth, its center, and its angular size. Then they run five fixed-point iterations to update scale, depth, and lateral position.

Lu: For dynamic objects, since the injected shapes already capture non-rigid motion and local rotation, they only need to recover a single global root orientation rather than one pose per frame.

Meng: And for appearance prediction, they use visibility attention bias to direct the model toward patches that actually see each voxel in the three dee representation <ref:2610.12388#pg1>. They also use rendering guidance during denoising and a loss function combining photometric and depth supervision.

Lalam: A big part of this is how they handle optional shape injection. If you have external geometry, they replace SAMthree dee’s coarse prediction with an occupancy grid inversion, which then conditions the pose denoising step.

Conclusion: Tom: So, to wrap up the GenIA paper, they show that this framework improves pose and appearance reconstruction across monocular, multi-view and dynamic inputs without retraining the foundation model. It’s about aligning it at three levels: geometry initialization, visibility reasoning, and differentiable rendering.

Jane: The results on ActionMesh show that GenIA ranks second to HiMoR on DAVIS training-view PSNR and δavg, while achieving the lowest reprojection error and highest CLIP-I across dynamic settings.

Lu: It suggests that by grounding SAMthree dee’s prior in observations this way, we get better color and lighting alignment than we see with models like SAMthree dee alone in a single view setting.

Meng: For practical applications, especially with dynamic scenes, this consistency across frames is important because per-frame SAMthree dee was temporally inconsistent. This work offers a more stable tracking correspondence for dynamic object reconstruction.

Lalam: Overall, GenIA is about taking the power of a large generative prior and making it reliable by forcing it to respect the physical constraints of what's actually in front of us at inference time.

More episodes

← Home