Bootstrapping a 4D LiDAR Annotation Tool from Video Foundation Models

arXiv:2608.25418 · cs.CV · Submitted 2026-08-26 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Bootstrapping a 4D LiDAR Annotation Tool from Video Foundation Models".

Jane: Progress in 4D LiDAR segmentation is currently bottlenecked by data scarcity, as assigning temporally consistent labels across sparse point cloud sequences is costly and difficult to scale.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So, let's start with the title and the authors of this paper, "Bootstrapping a 4D LiDAR Annotation Tool from Video Foundation Models." It’s pretty descriptive about what they are actually doing here.

Jane: I see that it’s led by Jihun Kim and his team at KAIST, which suggests a strong foundation in visual intelligence research for this kind of work. It’s interesting to see how they're combining their expertise with the capabilities of these large video models.

Lu: The authors bring a mix of deep computer vision and potentially some machine learning expertise that can handle the complexities involved in bridging 2D video understanding with 4D LiDAR data. I think their background is perfectly aligned for this kind of cross-modal research.

Meng: It's interesting how they are leveraging SAM2, which is a well-known model from the video domain, to tackle a problem that traditionally lives in the LiDAR space. It makes sense why they'd look to models that already have strong temporal understanding.

Lalam: I think it’s significant because it shows that we don't always need massive datasets of labeled three dee data to train powerful perception systems; sometimes, leveraging existing, well-trained models can give us a head start.

The paper's summary: Tom: Now for the actual summary of the paper, this research introduces LiDARSAM2 as a framework designed specifically to turn SAM2 into a source of supervision for the 4D LiDAR domain. Basically, they’re aiming to automatically generate temporally coherent labels directly from video masks.

Jane: That’s right, and I think the key innovation here is decoupling what needs to be segmented from how it's segmented by using an interactive prompting mechanism instead of relying solely on pre-labeled data.

Lu: What I find particularly compelling is the pipeline they use to go from 2D video masks to 4D LiDAR labels, which involves multi-view projection and a spatio-temporal 4D mask aggregation step. That sounds like a very clever way to enforce consistency across time.

Meng: From an engineering angle, that aggregation process sounds like it’s the part where they really have to handle the messy reality of aligning different views and ensuring those masks actually track correctly through time using ego-motion information.

Lalam: That spatio-temporal aggregation is what really makes this scalable; instead of labeling every single frame for every object, they are learning how to propagate a single initial click into a consistent track across the whole sequence.

The paper's improvements: Tom: Looking at the specific improvements they highlight in "Bootstrapping a 4D LiDAR Annotation Tool from Video Foundation Models," they emphasize two main things: strong labeling quality on SemanticKITTI and high downstream utility where models trained on these labels approach the accuracy of those trained on full manual annotation.

Jane: So, the paper claims that using their automatically generated labels can roughly double the mean Intersection-over-Union score compared to what standard SAM2 projection alone could achieve, which is a big quantitative claim.

Lu: The method itself involves a two-stage learning objective: first focusing on cross-modal alignment at the frame level using a Range-View representation, and then Stage two learns temporal propagation and object-level consistency across time. That’s how they adapt the video model to the LiDAR structure.

Meng: I'm interested in that Range-View projection idea; essentially treating LiDAR features as an RGB signal for SAM2 to process is a novel way to bridge those two modalities, but I have to wonder if that mapping introduces significant projection errors.

Lalam: The fact that they show these models reaching accuracy comparable to those trained on full manual annotation using just minimal prompts really speaks volumes about the potential efficiency gain this tool offers for everyone working in this space.

Conclusion: Tom: So, to wrap up on "Bootstrapping a 4D LiDAR Annotation Tool from Video Foundation Models," the main implication is that better training data curated automatically from foundation models can bring the scalability and generality of interactive segmentation into the 4D LiDAR domain.

Jane: It really seems like this provides a practical path toward large-scale, high-quality three dee scene understanding without needing extensive manual labeling efforts for every new task.

Lu: I think the core concept here is that you can take the general knowledge from a 2D video model and adapt its segmentation kernel to handle the unique spatio-temporal structure of LiDAR data through a tailored interface.

Meng: From my side, the practical implication is that this could drastically cut down on the time spent labeling data for autonomous driving perception models, which is where I see the biggest immediate impact for deployment readiness.

Lalam: I think this work shows that we can start building robust 4D scene understanding systems much faster by focusing on generating high-fidelity, temporally consistent pseudo-labels automatically.

Tom: Absolutely, it’s a solid piece of research demonstrating how to make interactive segmentation accessible even when the raw data is sparse. We’ll keep an eye on how this framework evolves for other domains.

Jihun Kim, Hyun-Kurl Jang, Hyemin Yang, Jinnyeong Yang, Hyeokjun Kweon, Kuk-Jin Yoon

KAIST Visual Intelligence Lab

cs.CV

Submitted: 2026-08-26

Updated: 2026-09-30

Importance score: 92/100

The gist: Progress in 4D LiDAR segmentation is currently bottlenecked by data scarcity, as assigning temporally consistent labels across sparse point cloud sequences is costly and difficult to scale.

Key concepts

SAM2
A 2D video foundation model used as a source of supervision. It excels at generating detailed segmentation masks from video streams, which are then leveraged to create initial temporal trajectories for LiDAR data. This allows the framework to generate labels automatically without requiring fixed, human-annotated datasets.
4D Mask Aggregation
The process of merging view-specific 2D masks into a consistent 4D sequence. This involves propagating masks across time using known ego-motion and fusing them via voxelization. This technique enforces temporal consistency, significantly increasing the density and quality of pseudo-labels derived from the video input.
Range-View (RV) Representation
A method used to bridge the gap between LiDAR data and SAM2's 2D image space. LiDAR features are projected into an image-like signal, allowing them to be treated as an RGB signal. This enables a cross-modal alignment stage where the framework learns how to map 3D LiDAR features into the 2D segmentation domain.
Two-Stage Learning Objective
A training strategy involving two distinct learning phases. Stage 1 aligns LiDAR frames with SAM2's representation at the frame level, while Stage 2 focuses on learning temporal propagation and object consistency across time for the final 4D segmentation task.

Terminology

Summary

Progress in 4D LiDAR segmentation is currently bottlenecked by data scarcity, as assigning temporally consistent labels across sparse point cloud sequences is costly and difficult to scale. This research introduces LiDARSAM2, a framework that leverages a 2D video foundation model (SAM2) to automatically generate scalable supervision for the 4D LiDAR domain. By treating SAM2 as a source of supervision rather than relying on fixed, human-annotated datasets, LiDAR-SAM2 aims to significantly reduce the annotation burden for 3D and 4D scene understanding by producing semantic and panoptic labels that approach full human annotation quality from minimal prompts.

The Core Idea: Leveraging Video Foundation Models for LiDAR Supervision

The central motivation is to turn a 2D video foundation model, SAM2, into a scalable source of supervision for the 4D LiDAR domain. The framework operates on the principle of decoupling what to segment from how to segment by using an interactive prompting mechanism. Instead of relying on manually labeled data, LiDAR-SAM2 is trained entirely from pseudo-labels that are automatically generated by SAM2 through multi-view projection and our spatio-temporal 4D mask aggregation. This approach positions the framework as a scalable labeling tool that substantially reduces the annotation burden for 3D and 4D scene understanding.

Data Generation Pipeline: From Video Masks to 4D LiDAR Labels

The process of generating temporally coherent LiDAR-level labels from SAM2 video masks involves several key steps:

  1. Applying SAM2 to multi-view RGB streams to obtain per-view temporal segmentation masks.

  2. Using these 2D binary masks as prompts for SAM2 to obtain a temporal mask trajectory across the sequence, denoted as a set of 4D LiDAR masks, where each initial click yields a consistent mask track across the entire sequence.

  3. Transferring each 2D binary mask into the 3D LiDAR domain via a calibrated projection function to obtain the per-frame 3D binary mask: m t,k 3D = Φ t(m t,k 2D).

  4. Performing spatio-temporal 4D mask aggregation to merge view-specific tracks and enforce consistency across time by propagating masks using known ego-motion and fusing them via voxelization. This aggregation substantially improves pseudolabel density while preserving quality, increasing the average number of segments from 22.12 without it to 73.84 on SemanticKITTI.

Modeling Strategy: A Two-Stage Learning Objective

To adapt SAM2’s video segmentation kernel to the spatio-temporal LiDAR structure, LiDAR-SAM2 employs a tailored modality interface and a two-stage learning objective, inspired by recent Vision-Language Models (VLMs):

  1. Stage 1 focuses on cross-modal alignment at the frame level. This involves mapping LiDAR frames into and out of SAM2’s 2D representation space using a Range-View (RV) representation, where LiDAR features are projected into an image-like signal: LiDAR features as an RGB signal. The objective here is defined by a segmentation loss between the predicted mask and the pseudo-label in the RV domain.

  2. Stage 2 focuses on learning temporal propagation and object-level consistency across time for 4D segmentation. After Stage 1, this stage trains the remaining SAM2 components (prompt encoder, memory attention, and mask decoder) to learn how to propagate segmentation consistently over time.

Evaluation and Utility: Performance Against Ground Truth

LiDAR-SAM2 was evaluated on SemanticKITTI using semantic segmentation metrics like mean Intersection-over-Union (mIoU) and panoptic metrics such as LiDAR Segmentation and Tracking Quality (LSTQ), which factorizes into an association score (Sassoc) measuring temporal instance consistency and a classification score (Scls). The results demonstrate that LiDAR-SAM2 labels lead to a substantial gain, roughly doubling the mIoU of naive SAM2, and bring both backbones close to models trained on full ground truth, despite using no human LiDAR annotation. Furthermore, the framework is evaluated as a labeling tool where simulated human interaction corresponds to roughly ten point prompts per frame for instance initialization. The resulting labels confirm that Object boundaries are clean and identities remain consistent across time, closely matching the underlying scene structure.

Conclusion

LiDAR-SAM2 successfully demonstrates that better training data, curated automatically from a foundation model, can bring the scalability and generality of interactive segmentation into the 4D LiDAR domain. The framework provides a practical path toward large-scale, high-quality 3D scene understanding by enabling interactive 4D LiDAR segmentation without manual labeling.

Improvements for AI systems

Here are the specific improvements that can be made to AI systems based on the LiDAR-SAM2 framework, along with what these improved systems can achieve:

  1. The ability to perform high-quality, interactive 4D LiDAR segmentation without requiring extensive manual annotation for every new domain or task.

  2. The creation of temporally coherent, dense semantic and panoptic labels across entire 4D point cloud sequences automatically from existing 2D video foundation models (like SAM2).

  3. The development of a scalable labeling assistant that drastically reduces the human annotation burden for 3D and 4D scene understanding tasks.

  4. The capability to generate high-quality training data for downstream 3D/4D perception models that approach the performance of models trained on full ground-truth supervision, even when only minimal human prompts are used during the initial interactive setup.

  5. The ability to adapt powerful 2D video foundation model segmentation kernels (SAM2) to operate directly on sparse, irregular 3D LiDAR point cloud geometry and spatio-temporal structures through a tailored modality conversion interface and a two-stage learning objective.

  6. The enhancement of 4D scene understanding systems by incorporating interactive prompting—allowing users to specify objects of interest via minimal point prompts—while maintaining consistent object tracks across the entire sequence.

  7. The improvement of 4D panoptic segmentation accuracy (measured by LSTQ, Sassoc, Scls) on datasets like SemanticKITTI by generating denser, more temporally consistent pseudo-labels than those produced by standard SAM2 projection alone.

This improved AI system (LiDAR-SAM2 framework) can be used to:

  1. Train autonomous driving perception models (e.g., for object detection, tracking, and scene understanding) using synthetic or semi-supervised data generated automatically from video footage without needing expensive LiDAR labeling teams.

  2. Enable rapid prototyping of new segmentation tasks in novel 3D environments by leveraging the general knowledge encoded in large 2D vision models, bridging the gap between 2D visual data and 4D spatial understanding.

  3. Create robust, temporally consistent object tracks for dynamic scenes (e.g., tracking vehicles or pedestrians across a sequence of LiDAR sweeps) by ensuring labels persist and grow beyond what is labeled in any single frame through spatio-temporal aggregation.

  4. Provide an efficient tool for labeling sparse, irregular point cloud data by turning sparse human interaction (a few clicks per object) into dense, high-fidelity 4D supervision, significantly accelerating the development cycle for complex 3D perception algorithms.

Sources

Related papers