OPERA: Object Perception Enhances Single-view 3D Reconstruction

The gist: Learnt object perception can significantly enhance 3D reconstruction by explicitly injecting perceptual signals from pretrained models into existing reconstruction pipelines to drive more accurate and semantically consistent results.

Introduction and Motivation

The relationship between object perception and reconstruction is well established in human vision, yet remains underexplored in computer vision The interplay between object perception and object reconstruction is a foundational principle in human vision Humans rely on high-level perceptual understanding, such as object semantics, structure, and affordances, to infer complete 3D shapes from partial observations Modern computer vision approaches to 3D reconstruction largely treat the problem as geometric or generative modelling These methods predominantly rely on data-driven priors or text/image conditioning, and lack an explicit mechanism to incorporate structured perceptual understanding of the object itself Single-view 3D object reconstruction is inherently ill-posed, requiring models to infer unseen geometry and synthesize novel viewpoints from limited input We argue that treating reconstruction purely as a generative or geometric problem is insufficient; instead, it should be grounded in object perception.

Methodology Overview

The proposed framework leverages foundation large-scale models, including vision-language models (VLMs) [30], largelanguage models (LLMs) [31], and foundation monocular depth estimation model [39] to extract perceptual knowledge guiding multi-view image generation The approach is model-agnostic and operates as a plug-and-play enhancement to existing reconstruction pipelines. The core of the method is the generation-perception alignment module (GPAM), which aligns the features learnt from the generation process with retrieved perceptual signals This integration is achieved by incorporating object perception with noise prediction, specifically estimating epsilon (i.e., estimation of ϵ(1:N)) to drive the generation towards the object’s perceptual knowledge.

Perception Modalities and Alignment

The framework demonstrates its effectiveness using two types of perceptual information 1. Semantic perception involves language descriptions, where Qwen2.5VL-7B generates a text caption from the input image y, which is then encoded into a 768-dimensional embedding s to serve as the perceptual signal 2. Geometric perception uses depth sensing via DepthAnything-V2 (ViT-L) [39], where intermediate feature maps are used to produce the depth perceptual signal s, which is observed to lead to better multi-view generation and later 3D reconstruction. The GPAM includes a Feature Aggregation module that aggregates intermediate activations from the U-Net into a compact multi-view representation and a Perception Mapping module that projects these aggregated features to the target perceptual space to produce the aligned feature u.

Training and Loss Functions

The training process involves minimizing a loss function L(s, u) which measures the similarity between perceptual signals s and generation-perception aligned features u For the semantic modality, the loss is defined as L(s, u) = 1 − coss(u) For the geometric modality, a loss based on similarity between generation-perception aligned features and depth features is used This geometric loss is decomposed into terms involving L2-std and Centered Kernel Alignment (CKA) losses for the front, front-left, and front-right views. The CKA loss is applied to the side views to encourage cross-view geometric consistency without requiring exact depth alignment as in the front view The training is performed by freezing both the multi-view image generative model (the U-Net) and the perception model(s) while training the GPAM for each perception modality.

Experimental Results

Experiments were conducted with two state-of-the-art single-view 3D reconstruction pipelines: Wonder3D [19] and Era3D [14] The results show that both semantic and depth knowledge individually reduce Chamfer Distance (CD) over the baseline, with their combination yielding the strongest result, such as a −9.0% reduction in CD for Wonder3D + semantic + depth The usefulness of perception for reconstruction is transferred across architectures, where combined perception guidance yields the lowest CD among all configurations Qualitative results show that the method recovers fine geometric details and corrects shape collapse on challenging objects.

Conclusion

We have shown that pre-trained perception models can serve as inference-time priors for single-view 3D reconstruction, steering a frozen single-view 3D reconstruction pipeline through a lightweight generation-perception alignment module Across two architecturally distinct pipelines, Wonder3D and Era3D, semantic and geometric perception priors each improve reconstruction quality, and combining them yields the strongest result The framework is model-agnostic and, in principle, extensible to other generative settings where pre-trained perception models exist and intermediate features admit gradient-based alignment. This work motivates further research on perception as a general prior for generative systems.

A Detailed Architecture Diagrams

Figure 3 illustrates the architecture of our method applied to Wonder3D with the semantic modality, showing how the Feature Aggregation module aggregates U-Net activations and the semantic Perception Mapping module produces the aligned feature u compared against caption embedding ss Figure 4 details the architecture applied to Wonder3D with the geometric (depth) modality, where aggregated RGB- and normal-stream features are mapped to u compared against DAv2 F3 target sd

A.1 Era3D Architectural Adaptations

The framework applied to Era3D shares the same conceptual structure as with Wonder3D, except for lighter aggregation width and unified forward pass The aggregation channel width is backbonedependent: Cagg = 384 for Wonder3D and Cagg = 128 for Era3D, whose Feature Aggregation module is deliberately lighter In both backbones, the geometric Perception Mapping module receives the channel-wise concatenation of the colour- and normal-domain aggregated features for the three supervised views (front, front-left, and front-right) The two backbones differ in how the two domains are produced: Wonder3D generates colour and normal views through two separate U-Net streams, whereas Era3D produces both domains in a single unified U-Net forward pass over a batch-concatenated 2N-view input

B Pseudo-code

Algorithm 1 presents the training code of our method, where the GPAM parameters γ are updated by minimizing L(s, u) Algorithm 2 presents the inference code, where noise prediction is guided by gradients derived from L(s, u)

C Semantic Perception Ablation

The two-step VLM+LLM approach consistently outperforms the single-step VLM-only baseline across all tested guidance weights Text captions provide a cleaner semantic signal because generating an intermediate caption encourages the VLM to summarise the object into higher-level concepts before re-encoding with T5

D Depth Perceptual Target

We select F3 for the depth perceptual signal s because it requires only a mild 1.

1. Begin single-view 3D object reconstruction by explicitly injecting perceptual signals from pretrained perception models capturing semantic and geometric information to drive the reconstruction of an object from its single image, which resolves ambiguities that classical and learning-based methods focus on mapping images to shapes using implicit representations or learnt priors often struggle with.

2. Enable model-agnostic integration by providing a plug-and-play manner enhancement, allowing the framework to be applied to existing pipelines such as both feed-forward reconstruction networks and generative approaches.

3. Enhance semantic consistency by leveraging language descriptions, specifically employing a two-step combination of a VLM (i.e., Qwen2.5VL-7B) and an LLM (i.e., T5) to extract a guidance signal that is less tied to low-level appearance than single-step VLM features, leading to better standalone reconstruction performance.

4. Improve geometric fidelity by utilizing depth sensing as a modality, where the loss function is structured as LL2std(front) + ωfront-right LCKA(front-right) + ωfront-left LCKA(front-left) with an optimal weight of w = 0.50, ensuring the model learns structural similarity in a viewpoint-invariant manner.

5. Achieve robust, combined reconstruction by leveraging multiple perceptual signals where combining semantic and geometric perception yields the strongest result, demonstrating that these modalities supply non-redundant corrections to the generative prior.