CLONE: Continuous Latent Optimization for Normal Estimation via 3D Gaussian Splatting

Summary

The paper proposes CLONE, a weakly supervised framework for single-image object surface normal estimation that does not require pixel-level normal ground truth. The core idea is to construct an image-geometry-image consistency loop that unifies explicit geometric representation with differentiable rendering. The paper states: "The core idea is to construct an image-geometry-image consistency loop that unifies explicit geometric representation with differentiable rendering, thereby enabling weakly supervised learning without normal ground truth."

The framework comprises four main components. First, a differentiable light interaction model with a learnable modulation kernel performs "a unified reparameterization of the 3DGS parameter space, establishing an explicit and stable mapping between 3DGS geometric parameters and surface normals and turning the photometric loss into an internal supervision signal. Second, a conditional single-step deterministic refinement network integrates denoising architectures with differentiable reprojection constraints to refine the initial normals, thereby adaptively recovering the high-frequency details erased by the inherently smooth Gaussian primitives. Third, a cross-domain gating fusion mechanism adaptively combines the two complementary normal estimates while imposing multi-view reprojection consistency and implicit geometric regularization, reconciling the geometrically consistent yet over-smooth 3DGS estimate with the detailed yet potentially geometry-inconsistent refinement. Finally, all components are jointly optimized under a unified photometric reprojection objective with geometric consistency regularizations, and a directional regularization aligns the learnable principal directions with the geometric normals, achieving an end-to-end optimization closed loop without relying on external normal labels."

The method is trained only on images paired with their corresponding 3D models, without any pixel-level normal annotations. The paper reports: "Trained only on images paired with their corresponding 3D models, without any pixel-level normal annotations, our method achieves state-of-the-art performance among weakly supervised methods and delivers accuracy competitive with fully supervised baselines."

The paper identifies limitations of existing weakly supervised methods: "Discriminative methods learn an image-to-normal mapping under proxy supervision signals built from pseudo-labels, cross-domain transfer or auxiliary alignment... the estimate's geometric fidelity is capped by the signals' own accuracy. Generative methods reconstruct 3D shape from a learned prior and read out normals as a by-product... the result reflects the prior's statistics, and when the prior and the observed image disagree, the estimate follows the prior. Each route therefore supplies only one of the two things the task requires: a geometry prior that provides the initial estimate or an image-derived signal that verifies it, and neither supplies both."

The main contributions are summarized as: "A unified framework that combines feedforward geometric priors with differentiable photometric verification through an image-geometry-image consistency loop, providing a self-supervising mechanism for weakly supervised normal estimation that corrects both proxy inaccuracy and prior deviation without external normal labels"; A high-fidelity differentiable illumination rendering pipeline with a learnable modulation kernel that turns the photometric loss into an internal supervision signal under weak supervision; A single-step deterministic refinement network that recovers high-frequency detail while preserving end-to-end differentiability, without external label data; and "A cross-domain gating fusion mechanism with multi-view reprojection consistency and implicit geometric regularization that reconciles geometric consistency with recovered detail, without relying on external normal labels."

The paper addresses four obstacles when adapting 3DGS to single-view weak supervision: "standard parameterization lacks a stable analytical mapping from covariances to normals without normal ground truth; native rendering pipelines sacrifice gradient accuracy for rasterization efficiency, providing insufficient fidelity for normal-level optimization; the intrinsic smoothness of Gaussian primitives erases high-frequency details, yet mainstream detail enhancement depends on normal ground-truth supervision; and naively fusing normals from geometric priors with those from detail enhancement breaks global consistency without ground-truth constraints on fusion boundaries."

The method defines each Gaussian primitive as G = µ, Σ, σ, kD, ωg, ξ, ψ, d, where µ is the Gaussian center, Σ is the covariance matrix, σ is opacity, kD is RGB diffuse albedo, ωg, ξ, ψ control frequency, spatial decay and phase shift of local light interaction, and d is a learnable local principal direction. The covariance matrix is parameterized using rotation-scale decomposition Σ = RSR T to ensure symmetric positive definiteness. Surface normals are derived via eigen-decomposition of the covariance matrix, where the eigenvector umin corresponding to the smallest eigenvalue gives the principal normal direction of the local surface, with sign disambiguation enforced by n3dgs T v > 0.

The illumination model is grounded in the Lambertian reflectance framework, with a learnable oscillatory modulation kernel K(p) = exp(−ξ∥p − µ∥22) cos((2π/ωg)d T(p − µ) + ψ) that captures high-frequency residuals from micro-geometric variations. The paper explains: "the explicit oscillatory form prevents the kernel from degenerating into a texture-fitting function, since it cannot reproduce arbitrary image patterns, and the Gaussian envelope ensures the modulation is localized, preserving the global dominance of the Lambertian term."

The radiance contribution of a single Gaussian primitive is formulated as Ci(x) = kD,i ⊙ max(0, n T fuse,i l) · Ki(p(x)), where l is the light direction fixed to the camera optical axis for training. The final pixel radiance is obtained by blending contributions from all Gaussians through the 3DGS rasterization process with cumulative transmittance.

The deterministic refinement network reformulates diffusion models from probabilistic generators to structural refinement operators. The paper states: Given n3dgs, Gaussian noise is first injected at a random time step t... One-step denoising is then performed using a noise prediction network ϵθ with time embedding to obtain the refined normal estimate. The one-step mapping is described as a noise-conditioned residual refinement, where the network learns to correct the initial normal using local structure information.

The cross-domain fusion mechanism introduces complementary feature representations: "The geometric feature Fgeo denotes the geometry-aware representation derived from the 3DGS-based prediction module... The texture feature Ftex denotes the appearance-aware intermediate feature extracted from the diffusion refinement network." The fused feature is expressed as Ffuse = Wg ⊙ Fgeo + Wt ⊙ Ftex, with weights obtained via softmax normalization ensuring Wg + Wt = 1 element-wise. The final fused normal is defined as nfuse = g ⊙ sg(n3dgs) + (1 − g) ⊙ ndiff, where g is a spatially adaptive gating weight predicted by a lightweight convolutional network.

The total loss is Ltotal = Lphoto + λscale Lscale + λnormal Lnormal + λdir Ldir, where Lphoto is the photometric consistency loss, Lscale is a scale regularization term preventing Gaussian primitives from degenerating, Lnormal is a normal consistency constraint encouraging the refined normal to stay close to the geometric normal in regions where the gating weight is small, and Ldir is a directional regularization aligning the learnable principal direction with the geometric normal.

Experiments are conducted on the Objaverse dataset for training (80k diverse 3D object models) and evaluated on Google Scanned Objects, Omniobject3D, and Wonder3D benchmarks. The paper reports: "under the weakly-supervised paradigm (no normal labels), CLONE achieves the best performance across all metrics. CLONE's MAE of 13.2◦ exceeds the strongest fully-supervised baseline Metric3D v2 (14.1◦, trained with ground-truth normals) by 0.9◦, and the strongest weakly-supervised baseline FE2E (14.3◦) by 1.1◦."

Ablation studies confirm the necessity of each core component: Eliminating the 3DGS geometric initialization module results in the sharpest performance decline, where MAE rises from 13.2◦ to 15.7◦ and Acc@11.25◦ drops by 6.9%. Removing the learnable modulation kernel elevates MAE to 13.9◦, removing cross-domain alignment also increases MAE to 13.9◦, removing gating fusion increases MAE to 13.6◦, and removing diffusion refinement increases MAE to 13.4◦.

Robustness tests show the method maintains stability under additive Gaussian noise, lighting variations, and viewpoint changes. The paper states: our method maintains exceptional stability: MAE increases by only 6.0◦ from σnoise = 0 to σnoise = 15, whereas the most affected baseline, Marigold-Normals, suffers a 13.3◦ MAE surge over the same noise range.

The paper acknowledges limitations including texture-geometry confusion, occlusion and slender structures, and non-Lambertian materials. The paper states: high texture gradients can mislead the diffusion branch, and specular reflections give rise to systematic errors. Furthermore, severe self-occlusion remains intractable due to the absence of explicit occlusion modeling. The paper concludes: "CLONE provides a new weakly-supervised paradigm for single-image single-object normal estimation by closing the loop between image observations and 3D geometry. It achieves competitive performance against fully supervised methods on synthetic benchmarks and demonstrates strong robustness to noise, lighting and viewpoint changes."

Based on the paper, here are the specific improvements I can make to AI systems and what the improved system can do:

1. Implement a self-supervising normal estimation pipeline using an image-geometry-image consistency loop

• Integrate 3D Gaussian Splatting (3DGS) as an explicit geometric representation with differentiable rendering to create a closed-loop optimization that verifies predicted geometry against the input image

• Replace the need for pixel-level normal ground truth with photometric reprojection loss as the primary supervision signal

2. Add a differentiable light interaction model with a learnable oscillatory modulation kernel

• Reparameterize the 3DGS parameter space to establish an explicit, stable mapping from Gaussian covariances to surface normals via eigen-decomposition

• Use the kernel K(p) = exp(-ξp-μ2)cos((2π/ω g)dT(p-μ)+ψ) to capture high-frequency micro-geometry variations beyond Lambertian reflectance, turning photometric loss into an internal geometric supervision signal

3. Integrate a single-step deterministic refinement network (conditional denoising U-Net)

• Reformulate diffusion models from probabilistic generators to structural refinement operators, avoiding gradient truncation inherent in multi-step DDPM

• Inject noise into the initial 3DGS normal, perform one-step denoising conditioned on fused geometric and texture features, and backpropagate photometric loss directly through the network

4. Implement a cross-domain gating fusion mechanism

• Fuse geometric normals (globally consistent but over-smooth) with refined normals (detail-rich but potentially geometry-inconsistent) using a spatially adaptive gating weight g ∈ [0,1]

• Apply joint channel-spatial attention with softmax normalization to combine 3D geometric priors and 2D appearance features, with a stop-gradient operation on the geometric branch to preserve end-to-end differentiability

5. Add multi-view reprojection consistency and geometric regularizations

• Use a scale regularization term L scale = Σ(λ min + γλ max) to prevent Gaussian primitive degeneration

• Add a normal consistency loss L normal = n 3dgs - n diff1 ⊙ (1-g) that adaptively relaxes constraints in texture-rich regions

• Include directional regularization L dir to align learnable principal directions with geometric normals

1. Estimate surface normals from a single RGB image without any pixel-level normal annotations

• Achieve state-of-the-art performance among weakly supervised methods (MAE 13.2° vs. 14.3° for the best competing weakly supervised baseline)

• Match or exceed fully supervised baselines (e.g., Metric3D v2 at 14.1° MAE) while using only RGB-3D model pairs for training

2. Recover high-frequency geometric details (thin structures, sharp edges, complex topologies) that pure 3DGS or generative methods erase

• Maintain consistent normals on slender objects (shoes, cactus branches, bamboo) where competing methods produce broken or discontinuous normals

• Accurately resolve part boundaries and junctions on complex objects (excavators, house models) without artifacts

3. Maintain robustness under challenging conditions

• Degrade gracefully under additive Gaussian noise (MAE increases only 6.0° from σ=0 to σ=15, vs. 13.3° for the worst baseline)

• Handle lighting variations (front, side, top) and viewpoint changes (0°, 45°, 70°) with minimal performance loss (MAE stays below 14.5° in all cases)

• Tolerate common image degradations like motion blur and JPEG compression

4. Operate with label efficiency

• Train on large-scale 3D asset libraries (e.g., Objaverse) without expensive per-pixel labeling

• Achieve competitive accuracy with only image-3D model pairs, reducing annotation costs by orders of magnitude compared to fully supervised approaches

5. Provide a unified framework that corrects both proxy inaccuracy and prior deviation

• The image-geometry-image loop ensures that errors in initial geometric estimates are corrected through photometric verification, avoiding the limitations of both pure discriminative and pure generative approaches

6. Achieve end-to-end differentiability with stable convergence

• Maintain a Gradient Transfer Ratio of 0.98 (vs. 0.12 for multi-step DDPM), ensuring gradients flow effectively through the entire pipeline

• Avoid training divergence (0% failure rate) through rotation-scale decomposition of covariance matrices, unlike direct covariance optimization which has 30% divergence risk

We propose CLONE, a Continuous Latent Optimization framework for Normal Estimation via 3D Gaussian splatting. The core idea is to construct an image-geometry-image consistency loop that unifies explicit geometric representation with differentiable rendering, thereby enabling weakly supervised learning without normal ground truth. Specifically, CLONE comprises four components. First, by introducing a differentiable light interaction model with a learnable modulation kernel, we perform a unified reparameterization of the 3DGS parameter space, establishing an explicit and stable mapping between 3DGS geometric parameters and surface normals and turning the photometric loss into an internal supervision signal. Second, the conditional single-step deterministic refinement network integrates denoising architectures with differentiable reprojection constraints to refine the initial normals, thereby adaptively recovering the high-frequency details erased by the inherently smooth Gaussian primitives. Third, the cross-domain gating fusion mechanism adaptively combines the two complementary normal estimates while imposing multi-view reprojection consistency and implicit geometric regularization, reconciling the geometrically consistent yet over-smooth 3DGS estimate with the detailed yet potentially geometry-inconsistent refinement. Finally, all components are jointly optimized under a unified photometric reprojection objective with geometric consistency regularizations in a fully differentiable pathway, and the directional regularization aligns the learnable principal directions with the geometric normals, achieving an end-to-end optimization closed loop without relying on external normal labels.