DeFM: Learning Foundation Representations from Depth for Robotics

summary

Video file (mp4)

The gist

The gist: DeFM, a self-supervised foundation model trained entirely on depth images, learns robust geometric and semantic representations that generalize across diverse robotics tasks without

In short

DeFM is a self-supervised foundation model trained entirely on 60 million depth images to learn robust geometric and semantic representations for robotics. It uses a DINOv2 style self-distillation objective to achieve generalization across diverse tasks, eliminating the need for task-specific fine-tuning. This allows for off-the-shelf deployment of powerful depth encoders on resource-constrained robotic systems.

Key concepts

Self-supervised Foundation Model
DeFM is a large model trained using self-supervision (DINOv2 loss) on massive amounts of unlabeled data (60M depth images). This allows the model to learn general, high-quality features from the data itself without needing human labels for specific tasks. It learns fundamental geometric and semantic understanding of depth that can be applied broadly.
DINOv2 Style Self-Distillation
This is the core training method where a 'student' model tries to mimic the predictions of a 'teacher' model on the same image. The loss function measures how closely their outputs match. This process forces the student to learn rich, consistent representations of depth, ensuring that features are both geometrically accurate and semantically meaningful across different views.
Input Normalization Strategy
To handle varying depth scales effectively, DeFM uses three input channels: Global Log-Scaled Depth (C1), Mid-Range Norm (C2), and Far-Range Norm (C3). Stacking these three channels creates the final input Xin. This strategy preserves metric depth information while making the model robust to different environmental depths.
Knowledge Distillation
This technique transfers knowledge from a large, complex 'teacher' model (DeFM) into a much smaller, faster 'student' model. By minimizing the difference between their outputs during training, the smaller student learns to perform tasks effectively using the powerful representations learned by the larger teacher. This makes DeFM practical for deployment on robots.

Terminology used across episodes

This episode discusses

The paper

DeFM: Learning Foundation Representations from Depth for Robotics · Read on arXiv

Manthan Patel, Jonas Frey, Mayank Mittal, Fan Yang, Alexander Hansson, Amir Bar3, Cesar Cadena1, Marco Hutter

Robotic Systems Lab (RSL), ETH Zurich, Switzerland · Stanford University, Stanford, USA · NVIDIA, Zurich, Switzerland

Depth sensors are widely deployed across robotic platforms, and advances in fast, high-fidelity depth simulation have enabled robotic policies trained on depth observations to achieve robust sim-to-real transfer for a wide range of tasks. Despite this, representation learning for depth modality remains underexplored compared to RGB, where large-scale foundation models now define the state of the art. To address this gap, we present DeFM, a self-supervised foundation model trained entirely on depth images for robotic applications. Using a DINO-style self-distillation objective on a curated dataset of 60M depth images, DeFM learns geometric and semantic representations that generalize to diverse environments, tasks, and sensors. To retain metric awareness across multiple scales, we introduce a novel input normalization strategy. We further distill DeFM into compact models suitable for resource-constrained robotic systems. When evaluated on depth-based classification, segmentation, navigation, locomotion, and manipulation benchmarks, DeFM achieves state-of-the-art performance and demonstrates strong generalization from simulation to real-world environments. We release all our pretrained models, which can be adopted off-the-shelf for depth-based robotic learning without task-specific fine-tuning. Webpage: https://de-fm.github.io/

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "DeFM: Learning Foundation Representations from Depth for Robotics".

Dev: The gist: DeFM, a self-supervised foundation model trained entirely on depth images, learns robust geometric and semantic representations that generalize across diverse robotics tasks without task-specific fine-tuning.

Rosa: First, who's behind it and why it matters.

Title and authors: Dev: Moving on to the title of this work, "DeFM: Learning Foundation Representations from Depth for Robotics." It really frames the entire effort around using depth as the foundation for robotics AI.

Rosa: Yeah, it clearly states that DeFM is a foundation model and that its training data is specifically depth images, which sets a very tight constraint on what it learns.

Taro: So when you hear that DeFM is learning foundation representations from depth, what does that actually mean in practice for someone building a robot system?

Dev: It means we’re taking a massive amount of raw depth information and teaching an AI model so that this model understands the three dee structure and the meaning of those structures <ref:2601.18923#pg1>.

Rosa: The implication is that instead of training a separate perception module for every single robotic task, you could use this one general model for everything from navigation to grasping.

Taro: Does this mean we can skip all that custom training when we move from simulation to the real world?

Dev: Not exactly skipping it entirely, but the paper shows that using frozen DeFM encoder features in downstream RL policies actually outperforms other frozen foundation model baselines like VAEs and DINOv3 in challenging real-world scenarios.

Rosa: That suggests a strong sim-to-real transfer capability, which is usually where those foundation models fall short when you hit the real world.

Taro: So the core idea is that the depth representation learned from sixty million images has this inherent generalization power across environments and sensors.

The paper's summary: Dev: Now let's look at how they built this, they describe a training process using DINOv2 style self-distillation to get those dense features in DeFM: Learning Foundation Representations from Depth for Robotics.

Rosa: So, they’re using that DINOv2 style self-distillation method on their curated dataset of sixty million depth images. The image-level objective is the LDINO loss between the student's output and the teacher's target distribution from class tokens <ref:2601.18923#pg1>.

Dev: And they enhance that with a patch-level objective called LiBOT loss to make sure it captures spatial details by checking consistency for randomly masked input patches against teacher targets.

Taro: It sounds like they’re using multiple ways to teach the model simultaneously: learning global structure from the image and local structure from individual patches.

Rosa: Exactly. And they prevented collapse in the feature space by using a KoLeo Regularizer, which is a repulsive term that keeps the feature diversity high during training.

Dev: So this whole setup—LDINO loss, LiBOT loss, and KoLeo Regularizer—is what builds this robust geometric and semantic representation model from scratch.

Taro: The dataset itself is quite diverse; it’s sixty million depth images made up of monocular depth estimation, synthetic data, and real data combined. That variety helps it learn more general features.

Rosa: And they used a specific input normalization strategy: Global Log-Scaled Depth, Mid-Range Norm, and Far-Range Norm. The final representation is just stacking those three channels into the Xin vector of three channels.

Dev: This input representation is designed to preserve metric depth while making sure the model remains robust across different scales relevant for things like near-field manipulation and long-range navigation.

The paper's improvements: Taro: I want to talk about the specific improvements they suggest. They really focus on making this massive pre-training useful for real robotics, moving beyond just getting good results on benchmarks.

Rosa: One major improvement is the distillation into compact models, ranging from three million to thirty million parameters, covering CNNs like ResNet and EfficientNet, and ViT-S transformers <ref:2601.18923#pg1>.

Dev: That size variation is key because it allows deployment on hardware like the NVIDIA Jetson AGX Orin for real-time inference in robotic systems.

Taro: And they show that this distillation leads to frozen models that outperform other frozen foundation model baselines in tasks like Habitat Point-Goal Nav.

Rosa: Plus, when fine-tuning is involved, the paper shows that fine-tuning actually improves results for all models tested, which is a nice result because it suggests the foundation model is a very strong starting point.

Dev: They also showed that this system can perform robust sim-to-real transfer by using frozen DeFM encoder features in downstream RL policies, outperforming VAEs and DINOv3 in challenging real-world scenarios.

Taro: The three-channel input normalization strategy, Xin =

C1, C2, C3: , is another key improvement because it ensures the model maintains metric depth while being robust across scales relevant for both dexterous manipulation and long-range navigation.

Rosa: So the takeaway here is that DeFM provides a powerful, general encoder that can be distilled into efficient models ready for deployment on resource-constrained hardware.

Conclusion: Dev: To wrap up, the paper "DeFM: Learning Foundation Representations from Depth for Robotics" shows that this self-supervised pretraining method is effective for learning robust geometric and semantic features from depth images.

Rosa: It’s a foundation model that generalizes well across diverse tasks, environments, and sensors without needing task-specific fine-tuning. This makes it an off-the-shelf depth encoder for perception and control.

Taro: What I want to add is that this system has the potential to streamline depth perception across a wide range of robotic systems, replacing many of the engineering-heavy pipelines that are still common today.

Dev: And because they distilled these models into small parameter counts, we can actually enable faster and more efficient RL training, which points toward a future where depth-based representations offer a simpler alternative to task-specific depth processing modules.

Rosa: Exactly. We've looked at how DeFM works, from the training methods to the final distilled models on various robots. It’s a solid piece of work for making deep learning more accessible in embodied AI research.

Taro: It really shows that a single frozen depth encoder could be a big step toward simpler, more general robotic systems.

More episodes

← Home