Selection, Not Fusion: Radar-Modulated State Space Models for Radar-Camera Depth Estimation

summary

Video file (mp4)

The gist

The gist The selection mechanism is the right place for radar to enter, introducing Radar-Modulated Selection (RMS), which achieves state-of-the-art performance on nuScenes by injecting radar into

In short

The paper introduces Radar-Modulated Selection (RMS), which integrates sparse radar data directly into Mamba's selective scan mechanism for depth estimation. RMS injects radar information into the model's step size and readout while keeping the core image processing unimpaired. This approach outperforms previous methods by focusing on selecting relevant radar data within the scan, proving that selection is a better place for multimodal fusion than fusing features outside the sequence operator.

Key concepts

Radar-Modulated Selection (RMS)
RMS modifies Mamba's selective scan by adding learned perturbations to its step size and readout. This allows radar to influence temporal credit assignment ($\Delta t$) and observation ($C_t$) directly within the model's internal selection process, ensuring radar only impacts the network where it provides accuracy improvements.
Multi-View Scan Pyramid (MVSP)
MVSP is a strategy used to manage computational load by matching the fusion operator to radar's reach at different scales. It uses three tiers: scene-wide RMS for coarse scales, windowed RMS for mid-scales, and FiLM modulation for fine scales. This ensures that computational resources are allocated optimally based on where radar data is most effective.
Selective Scan in Mamba
The selective scan is a mechanism within the Mamba architecture that determines which parts of the input sequence are most important to process. By injecting radar perturbations into this scan, RMS allows the model to selectively attend to and incorporate relevant radar signals during its processing steps, rather than just concatenating features afterward.
Linear Cost Fusion
The paper emphasizes coupling modalities at a linear cost. This means the integration of radar data with image data is done in a way that does not significantly increase the computational complexity beyond what is necessary for the selective scan itself, maintaining efficiency while achieving multimodal fusion.

Terminology used across episodes

This episode discusses

The paper

Selection, Not Fusion: Radar-Modulated State Space Models for Radar-Camera Depth Estimation · Read on arXiv

Zhangcheng Hou, Tomoaki Ohtsuki

Keio University

Radar camera depth estimation turns a very sparse but metric radar signal into a dense depth map. Existing methods combine radar and image features outside the sequence operator of the backbone, through concatenation, confidence gating, sparse supervision, or graph based extraction. Mamba variants for multimodal fusion also leave the selection mechanism itself unimodal. We argue that selection is the right place for radar to enter. We introduce Radar-Modulated Selection (RMS), which injects radar inside the selective scan of Mamba. Radar adds learned perturbations, initialised at zero, to the step size Δ and the readout C. The input projection B and the state matrix A never see radar. At initialisation the block equals a pretrained image Mamba, so radar gains influence only where it improves accuracy. The construction also couples the two modalities at linear cost and falls back to the image backbone when radar is missing. We deploy RMS in a Multi-View Scan Pyramid (MVSP) that matches the fusion operator to the reach of radar at each scale. On nuScenes, SemoDepth reduces MAE by 34.0%, 29.9%, and 29.9% over the previous best within 50, 70, and 80 m. It also has the lowest single frame latency among methods with a public release (26.8 ms). A feature blend placed before the scan adds no accuracy on top of RMS, which supports the view that selection inside the scan can replace fusion outside it. When twice as many spurious radar returns as true ones are injected, the MAE@80 of SemoDepth rises by 1.4%, against 33.6% for a Mamba baseline that fuses radar before the scan.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Selection, Not Fusion".

Jane: The gist The selection mechanism is the right place for radar to enter, introducing Radar-Modulated Selection (RMS),

Tom: First, who's behind it and why it matters.

Title and authors: Tom: We started with the paper, "Selection, Not Fusion: Radar-Modulated State Space Models for Radar-Camera Depth Estimation." The authors are Zhangcheng Hou from Keio University.

Jane: They are tackling the challenge of turning sparse radar signals into dense depth maps using only a camera and radar sensor. The title itself suggests they aren't just combining features, but fundamentally changing the selection process inside the model.

Lu: It’s about injecting radar directly into Mamba’s selective scan mechanism > Instead of adding it outside the backbone or as a separate fusion step, they argue that it belongs where token-level selection happens >

Meng: So this means if you use an architecture like Mamba, which uses selection to decide what information to keep, you modify the selection criteria itself using radar data.

Lalam: Right. It’s about changing the input-dependent step size delta and readout C parameters with radar signals while leaving the image-only projections B and state dynamics A untouched >

Tom: That’s a big point because it means they are modifying how information is stored and propagated rather than just how they combine it >

Jane: So, what’s the core mechanism they are proposing for this radar modulation? How do we make sure we understand this concept without getting lost in the math?

Tom: They introduce Radar-Modulated Selection, or RMS, which is a minimal and principled way to inject radar into Mamba’s selective scan > They achieve this by adding zero-initialised perturbations to the step size delta and readout C while leaving the input projection B and state dynamics A image-only >

Lu: It’s important that they state it's minimal because it's a principled way to do it, not some arbitrary hack >

The paper's summary: Tom: In short, the paper argues that existing methods combine radar and image features outside the backbone’s sequence operator. They call this "selection, not fusion."

Jane: So they are arguing that putting radar inside the backbone’s sequence operator for selection is where it should be >

Lu: They propose Radar-Modulated Selection, RMS, which injects radar inside Mamba's Gu and Dao two thousand twenty-three selective scan > This fundamentally changes how information is stored and propagated rather than merely how they combine it > <ref:2605.11840#pg2,propose Radar-Modulated Selection, RMS, which injects radar inside Mamba's Gu>

Meng: So the gist is that if you want better performance, don't just fuse features; change the core selection logic itself.

Lalam: It’s about controlling what each token retains and emits based on radar input via delta t and C > They control what gets stored and emitted at each step >

Tom: Exactly. And they show this construction is equivalent to a pretrained image-only Mamba at initialisation, which ensures radar only influences the model where it improves accuracy >

The paper's improvements: Jane: So that’s a strong claim about influencing the model only where it helps, and what are the concrete results they found when they tested this RMS approach?

Tom: They showed state-of-the-art performance on nuScenes by reducing MAE by thirty-four point zero percent, twenty-nine point nine percent, and twenty-nine point nine percent over previous bests at ranges of zero to fifty meters, zero to seventy meters, and zero to eighty meters >

Lu: That’s a substantial reduction in error across those specific distance ranges; they aren't just tweaking it slightly.

Meng: And they also tested out-of-scan feature blending on top of RMS, and they found that it added no accuracy on top of RMS > So their finding is that in-scan selection can replace out-of-scan fusion entirely >

Tom: That means the performance gain comes purely from the in-scan selection mechanism, not from some extra feature blending techniques >

Conclusion: Jane: So to wrap up, it sounds like this paper is proposing a new way to integrate radar data into sequence models that focuses on selection rather than fusion.

Tom: Exactly. The main implication is that we can achieve state-of-the-art depth estimation on nuScenes by replacing out-of-scan fusion with in-scan selection, and they achieved this with Radar-Modulated Selection >

Lu: It’s a principled way to do it because it ties the radar input directly to temporal credit assignment via delta t and observation via Ct >

Meng: And they managed the computational cost by using the Multi-View Scan Pyramid to match fusion operators to radar’s reach at different resolutions >

Lalam: They are leaving us with a system that is competitive in terms of inference efficiency, achieving the lowest single-frame latency among methods with public releases at twenty-six point eight milliseconds >

Jane: So for the listener, this means we have a method that delivers high accuracy without needing expensive LiDAR on every vehicle.

Tom: It’s a strong result because they didn't just claim an MAE reduction of thirty-four point zero percent and twenty-nine point nine percent > We see the mechanics behind it >

Lu: The paper "Selection, Not Fusion: Radar-Modulated State Space Models for Radar-Camera Depth Estimation" provides a clear direction on how to move multimodal learning forward by focusing on selection over representation >

More episodes

← Home