Distilling Routed 3D Privilege for Spatial Reasoning in Vision-Language Models

summary

Video file (mp4)

The gist

The gist The gist GPD, an on-policy self-distillation framework, augments GRPO with an OPSD teacher conditioned on question-routed depth, semantic, and BEV textual cues together with the reference

In short

GPD augments existing spatial reasoning methods by using geometry-privileged self-distillation during training. It combines question-routed 3D cues (depth, semantic, BEV) with the correct answer to guide a student model. This allows the model to learn geometric reasoning while keeping inference RGB-only, outperforming other methods on spatial benchmarks.

Key concepts

GPD (Geometry-Privileged Distillation)
This is a new training framework that combines outcome-based policy optimization (GRPO) with distillation. It makes geometric evidence the 'privilege' during self-distillation by feeding the teacher model specific 3D cues related to the question and answer, helping the student internalize spatial reasoning.
Question-Conditioned Routing
This is an offline process where, for every training example, only the subset of scene information relevant to that specific question is selected. This ensures that the teacher receives a compact and focused 3D context, preventing the model from being overwhelmed by irrelevant scene details during distillation.
Geometry-Privileged Self-Distillation (OPSD)
This is the core training mechanism where a privileged teacher model re-scores incorrect trajectories using both RGB input and the question-routed 3D cues. This selective distillation focuses the learning signal on where the student model made spatial errors, forcing it to correct its geometric understanding.
RGB-only Inference
The final deployed model only requires standard RGB images as input for making predictions. GPD achieves this by internalizing the necessary geometric knowledge during the training phase through the privileged distillation process, avoiding the need for expensive 3D processing at runtime.

Terminology used across episodes

This episode discusses

The paper

Distilling Routed 3D Privilege for Spatial Reasoning in Vision-Language Models · Read on arXiv

Hongxing Li, *, *Yixin Li, *Dingming Li, Zixuan Wang, Yuchen Yan, Wenqi Zhang, Weiming Lu

Zhejiang University

Spatial reasoning remains a persistent weakness of vision-language models (VLMs), because RGB inputs do not directly provide geometric evidence. Existing remedies either inject 3D into the model at inference, paying architecture and latency costs, or train with outcome rewards that supervise only the final answer. Spatial errors originate in perception: a misjudged depth or direction can be corrected only by the scene's true geometry, which the 3D-scanned sources of spatial training corpora already provide. We propose GPD (Geometry-Privileged Distillation), which makes geometric evidence the privilege in on-policy self-distillation (OPSD). For each question, depth, semantic, and bird's-eye-view (BEV) cues are rendered as compact text and routed to the teacher alongside the reference answer; a privileged KL, applied only to incorrect trajectories, augments GRPO, and the deployed model remains RGB-only. On the 4B backbone, GPD achieves 57.1 on VSI-Bench and 37.6 average across MindCube, SPARBench, MMSI-Bench, and ViewSpatial, outperforming both GRPO and answer-privileged OPSD across spatial reasoning benchmarks. Ablations confirm the complementarity of 3D and answer privilege, the advantage of question-conditioned routing over full-context injection, and the benefit of restricting distillation to incorrect trajectories.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Distilling Routed 3D Privilege for Spatial Reasoning in Vision-Language Models".

Jane: The gist The gist GPD, an on-policy self-distillation framework, augments GRPO with an OPSD teacher conditioned on question-routed depth, semantic,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: To recap, this paper introduces GPD, which is an on-policy self-distillation framework that augments GRPO with a teacher conditioned on question-routed depth, semantic cues and bird’s-eye-view views alongside the reference answer.

Jane: The core idea they are pushing is that spatial errors come from perception—like misjudging depth or direction—and these can only be corrected by the scene’s true geometry, which their three dee-scanned sources provide for training <ref:2610.12355#pg1,by the scene’s true geometry, which>.

Lu: They construct a privilege by first rendering depth, semantic segmentation masks and labels using something like Grounded-SAM-two if they don't have aligned three dee annotations available <ref:2610.12355#pg2>.

Meng: Then they do question-conditioned routing offline to select a subset of training data relevant to the specific question. This forms the scene context for the teacher, which is then combined with the reference answer into a privileged context.

Lalam: During training, the student model sees only RGB and its question, while the teacher sees all that extra three dee information along with those routed cues <ref:2610.12355#pg1>.

Tom: The optimization objective combines outcome-based GRPO loss with this privilege-guided distillation applied selectively to incorrect trajectories, which is what they call L˜OPSD.

Jane: So, the student learns from the teacher's guidance only when it’s on a trajectory that was wrong, using that extra geometric context to correct its spatial understanding.

Tom: The results are pretty solid; on the 4B backbone, GPD achieves fifty-seven point one on VSI-Bench and an average of thirty-seven point six across five other benchmarks like MindCube and ViewSpatial <ref:2610.12355#pg2,on the 4B backbone, GPD achieves 57.1 on VSI-Bench>.

Lu: The authors confirm that this approach shows the complementarity between the training privilege from three dee data and the answer privilege, which is really interesting for how we structure supervision in these models <ref:2610.12355#pg1>.

Meng: They also showed that question-conditioned routing is better than just injecting full context because it's more targeted.

Lalam: And they found that restricting distillation only to incorrect trajectories helps a lot with the overall performance lift.

Conclusion: Tom: So, looking at the whole paper "Distilling Routed three dee Privilege for Spatial Reasoning in Vision-Language Models," what this means is that we can move away from needing heavy, expensive three dee architecture just to get better spatial understanding <ref:2610.12355#pg1,Distilling Routed 3D Privilege for Spatial Reasoning in Vision-Language Models>.

Jane: Instead, we can use training signals that are derived from geometry—depth and BEV cues—as a privilege for distillation.

Lu: The implication is that an RGB-only model can learn to reason spatially if it's trained with this specific, question-routed geometric supervision during the on-policy self-distillation process.

Meng: For practical deployment, this means we don't have to worry about adding latency or complex three dee encoders at inference time because the reasoning is already baked into the model from training <ref:2610.12355#pg1>.

Lalam: The authors found that this method improves overall accuracy over both GRPO and answer-privileged OPSD across different scales and benchmarks.

Tom: It seems they’ve successfully turned geometric evidence into supervision that an RGB-only model can internalize during training, which is what the paper is really about.

Jane: It shows that we don't necessarily need explicit three dee methods or just outcome rewards; we can use targeted three dee signals as a training privilege to improve spatial reasoning in vision models <ref:2610.12355#pg1>.

More episodes

← Home