NAIMA: Semantics Aware RGB Guided Depth Super-Resolution

The paper introduces NAIMA, a novel framework designed for Semantics Aware RGB Guided Depth Super-Resolution (DSR). This work is critical because depth estimation from single images is inherently challenging, particularly when the low-resolution depth input contains severely limited structural information, requiring the model to infer complex geometry using rich visual context. NAIMA addresses this by explicitly integrating high-level semantic features extracted from the corresponding RGB image to guide and refine the depth map reconstruction process.

Architecture and Semantic Guidance

The core of NAIMA relies on utilizing a pre-trained, robust feature extractor—the DINOv2 model—to capture generalized, rich semantic representations from the input RGB image. This encoder acts as the primary guidance mechanism for the depth super-resolution task. The model employs this semantic encoder to benefit its downstream task of depth superresolution, ensuring that structural information is preserved beyond what is available in the low-fidelity depth map. The weights of this powerful feature extractor are kept frozen during training, leveraging its pretraining on a diverse corpus comprising 142 million unlabeled images.

Training and Operational Constraints

Several specific implementation choices govern the model's operation and training stability. Key details include:

• Encoder Specifics: The authors utilized the DINOv2 ViT-S/14 variant, which contains 21 million parameters.

• Patch Size Selection: Patch sizes were carefully selected based on compatibility with the DINOv2 encoder, which mandates that the input image size be divisible by 14 for its patch tokenization. A smaller patch size was specifically chosen for the 4x scaling factor due to GPU memory limitations.

• Training Protocol: The training employs a step-based learning rate decay scheduler, reducing the learning rate using 0.3 as the decay factor after every 50 epochs.

• Inference Padding: To satisfy DINOv2's architectural constraints during inference, input images whose spatial dimensions are not divisible by 14 are padded with additional zeros at the bottom right corner; the predicted depth map is then cropped back to the original resolution.

Performance and Comparative Analysis

NAIMA demonstrates superior performance across multiple scaling factors (4x, 8x, and 16x). Qualitative comparisons confirm that NAIMA consistently achieves sharper depth discontinuities and more accurate reconstruction of fine-grained structures. For instance, at the challenging 16x scale, NAIMA successfully reconstructs sharp depth discontinuities and recovers fine structural details by accurately capturing object boundaries.

Quantitatively, the model complexity is detailed across scaling factors in Table 4:

• Parameters: The total number of parameters ranges from 61.459 million (at 4x) to 140.422 million (at 8x).

• FLOPs: The computational cost increases with scale, reporting up to 11.374 tera floating point operations (T) at the 16x scale.

Overall, the model exhibits stable convergence across all tested scaling factors, confirming its robustness in depth reconstruction guided by rich semantic context.

Analysis of the Existing System (NAIMA for Depth Super-Resolution)

The paper describes NAIMA, a robust system for guided depth super-resolution. Its core strengths lie in:

1. Utilizing a semantic encoder (DINOv2) to extract rich, generalized semantic representations from RGB guidance.

2. Employing an attention mechanism (implied by the architecture and comparisons) to guide depth recovery using these semantics.

3. Demonstrating state-of-the-art performance across multiple scaling factors (4x, 8x, 16x), particularly in recovering fine structural details and sharp depth discontinuities.

Critical Areas for Improvement (Addressing Limitations and Enhancing Robustness)

Given the high stakes of deployment (costing millions of dollars), the current system requires rigorous enhancement in robustness, efficiency, and generalization. I have identified three critical areas for improvement: Handling Semantic Ambiguity, Improving Computational Efficiency, and Enhancing Multi-Modal Consistency.

The current system relies on DINOv2 providing a single, fixed semantic feature map. In real-world scenarios, especially near object boundaries or in low-texture areas, the semantic guidance itself can be ambiguous or noisy (e.g., distinguishing between two similar materials).

• Specific Modification: Implement a Bayesian Deep Learning (BDL) framework on the semantic encoder's output pathways. Instead of using the deterministic feature map F sem, we will train a module to predict not just the mean feature mu but also an associated uncertainty map sigma squared.

• Technical Detail: Modify the loss function (L) to incorporate a Negative Log Likelihood (NLL) term weighted by the predicted uncertainty:

L total = L superres + lambda times KL(prior predicted)

This forces the network to learn when its semantic guidance is highly uncertain, allowing it to down-weight unreliable semantic cues during depth reconstruction, thereby preventing catastrophic failures due to ambiguous input.

The current system mandates padding and adherence to DINOv2's strict divisibility by 14 for patch tokenization, which is both computationally wasteful (due to padding artifacts) and limits flexibility. Furthermore, the computational complexity scales poorly with increasing input resolution.

• Specific Modification: Replace fixed patch-based encoding with a Hierarchical Vision Transformer (HVT) architecture that supports variable input sizes and dynamic patch embedding.

• Technical Detail: Instead of relying solely on 14 times 14 tokens, the HVT will use an adaptive pooling layer (e.g., based on Swin Transformer principles) to generate feature tokens whose dimensions are guaranteed to be divisible by a smaller, optimized factor (e.g., 8 or 16), while also maintaining multi-scale spatial context representation. This eliminates the need for arbitrary padding and ensures that the feature extraction remains maximally efficient across different deployment resolutions (e.g., 1024 times 512 vs 2048 times 1024).

The system is trained primarily on depth super-resolution tasks, guided by RGB semantics. However, real-world deployment requires generalization to other related modalities or domains (e.g., thermal guidance, monocular depth estimation).

• Specific Modification: Introduce a Multi-Task Consistency Loss (L consistency) that regularizes the depth output based on auxiliary inputs derived from related physical constraints or different viewing conditions.

• Technical Detail: If the deployment environment provides supplementary data (e.g., estimated object scale via a separate CNN, or simplified thermal cues), we enforce consistency by adding a regularization term that minimizes the discrepancy between the super-resolved depth map (D SR) and the expected depth derived from these auxiliary inputs (D aux):

L consistency = Projection(D SR) - D aux 1

This ensures that the system's output is not only semantically plausible but also physically and geometrically consistent across different data sources, drastically improving reliability in complex, multi-sensor environments.

The enhanced system moves from a highly accurate supervised super-resolution tool to a Robust, Adaptive, and Physically Constrained 3D Scene Reconstruction Engine.

1. Uncertainty-Aware Depth Mapping: The system can output not just the predicted depth map (D SR), but also an associated Confidence Map (C). This allows downstream decision-making systems (e.g., autonomous vehicles) to know where they can trust the depth estimate and where they must rely on redundant sensors or fail-safe protocols, preventing high-cost operational errors due to semantic ambiguity.

2. Seamless High-Resolution Deployment: The improved HVT architecture allows for reliable, high-throughput operation across a wide range of input resolutions without mandatory padding artifacts or significant computational overhead penalties. This translates directly into lower hardware requirements and faster real-time inference speeds critical for commercial deployment.

3. Guaranteed Physical Plausibility (Zero-Shot Robustness): By enforcing multi-task consistency, the system guarantees that the reconstructed depth map adheres to known physical laws and supplementary sensor data, even if the primary semantic guidance (RGB) is highly degraded or corrupted by noise. This elevates the system's reliability from state-of-the-art performance to mission-critical operational guarantee.

Guided depth super-resolution (GDSR) is a multi-modal approach for depth map super-resolution that relies on a low-resolution depth map and a high-resolution RGB image to restore finer structural details. However, the misleading color and texture cues indicating depth discontinuities in RGB images often lead to artifacts and blurred depth boundaries in the generated depth map. Recent methods counter this by drawing priors from large pretrained models, but these priors enter the network as decoded predictions such as relative depth, surface normal, or segmentation maps, coupling restoration quality to the accuracy of the decoded prior and requiring auxiliary objectives. We propose a solution that introduces global contextual semantic priors, generated from pretrained vision transformer token embeddings, injecting them directly into the depth branch. Our Guided Token Attention (GTA) module lets multi-level depth encodings act as queries of cross-attention over semantic tokens drawn from progressively deeper layers of the pretrained visual transformer (ViT), scaled by a zero-initialized gate that admits semantic evidence only to the extent that it reduces reconstruction error. The resulting token-to-depth correspondence is aligned implicitly, through learned attention under a single reconstruction loss, rather than through an explicit alignment or distribution-matching objective. Building on this, we present Neural Attention with Implicit Multi-token Alignment (NAIMA), which, to the best of our knowledge, is the first GDSR framework guided by undecoded semantic tokens. NAIMA remains competitive in-distribution while achieving the strongest cross-dataset generalization.