mmWave Radar Aware Dual-Conditioned GAN for Speech Reconstruction of Signals With Low SNR

arXiv:2602.22431 · cs.SD, cs.LG · Submitted 2026-08-15 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "mmWave Radar Aware Dual-Conditioned GAN for Speech Reconstruction of Signals With Low SNR".

Jane: The paper was written by Jash Karani, Adithya Chittem, Deepan Roy and Sandeep Joshi from BITS Pilani.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper discussion segment 1: Tom: Welcome back to the show, everyone! We've got a fascinating paper to dig into today, and it's called "mmWave Radar Aware Dual-Conditioned GAN for Speech Reconstruction of Signals With Low SNR." Jane, I have to say, the title alone is a mouthful, but the idea behind it is honestly kind of wild.

Jane: It really is, Tom. So the basic idea here is that they're using millimeter-wave radar — the same kind of technology that's in your car for adaptive cruise control — to actually pick up human speech. Not through a microphone, but by detecting the tiny vibrations of a speaker's diaphragm, or even a piece of aluminum foil sitting nearby.

Tom: Right, and that's the part that blows my mind. The radar can see these vibrations through a glass wall. So you could be in one room, and the radar is in another, and it's picking up the vibrations of someone talking. But here's the catch — the signal it captures is really, really bad. We're talking about a signal-to-noise ratio of negative five to negative one decibels.

Jane: And for our listeners who aren't audio engineers, that means the actual speech signal is buried way below the noise floor. It's like trying to hear a whisper in the middle of a rock concert. On top of that, the radar only captures a very narrow band of frequencies — basically just the lowest part of the human voice, up to about one kilohertz.

Tom: So you've got this incredibly noisy, band-limited signal, and you need to reconstruct full, intelligible speech from it. That's the problem this paper tackles. And the team behind it is from BITS Pilani in India — Jash Karani, Adithya Chittem, Deepan Roy, and Sandeep Joshi. They've built a system they call RAD-GAN, which is a type of generative adversarial network.

Jane: And for anyone who hasn't heard of GANs before, think of it like a forger and an art critic. The forger, which is the generator, tries to create realistic speech from the radar data. The critic, which is the discriminator, tries to tell the difference between the generated speech and real, clean speech. They keep going back and forth, and eventually the forger gets so good that the critic can't tell the difference anymore.

Tom: That's a great way to put it, Jane. But what's really clever here is that they didn't just use a standard GAN. They added some custom pieces to handle the specific challenges of radar data. And I think that's what we should dig into next — how they actually made this work when so many other approaches have struggled.

Jane: Absolutely. And the fact that they're doing this with a relatively small dataset, without any pretrained models, and without data augmentation — that's a pretty big deal. Let's talk about how they pulled that off.

Paper discussion segment 2: Tom: So, Jane, we've established that this RAD-GAN system is trying to turn terrible radar signals into intelligible speech. But what makes their approach different from just throwing a standard GAN at the problem?

Jane: Great question. The first thing they did was look at what the radar actually captures well. They found that the most reliable spectral energy is below one kilohertz — about seventy-five percent of it in their first task, and fifty percent in the second. Everything above that is mostly noise. So instead of trying to learn from that noisy high-frequency garbage, they decided to focus on the clean low-band information.

Tom: So they're essentially saying, "Let's only trust the bottom of the spectrum, and then use that to hallucinate the rest." That's a smart move. But how do they actually train the model to do that hallucination?

Jane: They use a two-stage training strategy. In the first stage, they take clean speech, clip it down to that one-kilohertz band, and train the generator to reconstruct the full bandwidth. This is like teaching the model what a voice should sound like when you only give it the low notes. They do this without any adversarial training — just pure reconstruction losses.

Tom: And then the second stage?

Jane: In the second stage, they fine-tune on the real, noisy radar data. But here's where it gets interesting — they don't just feed the noisy radar mel-spectrogram into the generator. They also have a separate module called WaveVoiceNet, which is a pretrained enhancer, and they fuse the two together using something they call a Residual Fusion Gate.

Tom: That fusion gate sounds like the secret sauce. Can you break that down for us?

Jane: Sure. So you have two inputs: the noisy mel-spectrogram, which is the raw radar data, and the WaveVoiceNet-enhanced mel-spectrogram. The fusion gate looks at both of them and decides, frame by frame, which one to trust more. If the WaveVoiceNet output looks reliable, it boosts that. If it looks like it's adding artifacts, it falls back to the noisy input. It's like having a smart mixer that automatically adjusts the levels between two microphones.

Tom: That's clever. And they also added a custom discriminator called the Multi-Mel Discriminator, which works on mel-spectrograms instead of raw waveforms. That gives the generator feedback on whether the spectral structure looks realistic, not just whether the waveform looks right.

Jane: Exactly. And the results speak for themselves. On their weighted score, which combines multiple quality metrics across both tasks, they hit zero point three three three. The best previous approach, WaveVoiceNet, only got zero point two six zero. And a plain HiFi-GAN, which is a strong baseline, got zero point two eight eight. So they're beating the state of the art by a meaningful margin.

Tom: That's a solid improvement. But I'm curious — how much compute did this take? Because a lot of these speech reconstruction models are incredibly expensive to train.

Jane: That's actually one of the most impressive parts. The whole fine-tuning phase took about fourteen hours on a single NVIDIA A6000 GPU. The pretraining was about six hours. So the entire pipeline is under a day of training on one GPU. That's really accessible compared to some models that need clusters of GPUs for weeks.

Tom: That's remarkable. And it makes me wonder — what does this mean for real-world applications? Let's bring in Lu and Meng to get their take on that.

Paper discussion segment 3: Lu: Thanks, Tom. I've been listening to you two break this down, and I think the implications here go way beyond just making a better speech reconstruction model. This paper is essentially demonstrating a new way to sense the world. Radar doesn't need light, it doesn't need a direct line of sight, and it can penetrate non-metallic materials. That opens up some really interesting possibilities.

Jane: Lu, I love where you're going with this. What kind of applications are you thinking about?

Lu: Well, consider search and rescue. First responders could use a radar system to detect if someone is alive and talking under rubble after an earthquake or a building collapse. The radar could pick up the vibrations of their vocal cords, and this system could reconstruct what they're saying. That's not science fiction — that's using technology that already exists in a new way.

Meng: That's a compelling use case, Lu. But as an engineer, I have to ask about the practical constraints. The paper mentions this was tested through a glass wall, which is a pretty specific scenario. How does this generalize to other materials, or to longer distances? And what about the latency — can this run in real-time?

Tom: Those are exactly the questions I was hoping you'd ask, Meng. The paper doesn't address real-time deployment, but the fact that the model is relatively small — about eighty-seven million parameters — suggests it could run on edge devices. The authors mention future work on latency and model compression for edge inference.

Lu: And on the generalization question, I think that's where the two-stage training really helps. By pretraining on clean speech and then fine-tuning on radar data, the model learns a general mapping from low-band to full-band speech. That should transfer reasonably well to different radar configurations, as long as the low-band signal is still reliable.

Meng: I'd like to see more testing on that, though. The paper evaluates on two tasks — direct diaphragm vibration and vibration through an aluminum foil reflector. Those are both pretty controlled scenarios. Real-world environments have moving objects, multiple speakers, and all kinds of interference. But for a first demonstration, the results are genuinely impressive.

Jane: And that's the thing, Meng — this is a proof of concept. It shows that with the right architecture and training strategy, you can extract intelligible speech from incredibly degraded radar signals. The fact that they did it with a limited dataset and no pretrained modules makes it even more compelling.

Lu: I also think about the privacy implications. If radar can pick up speech through walls, that has surveillance applications — both good and bad. On the good side, it could help law enforcement in hostage situations. On the concerning side, it could be used for unauthorized eavesdropping. This is a technology that we should think about carefully as it develops.

Tom: That's a really important point, Lu. The same technology that helps first responders could also be misused. It's a conversation we need to have as a society. But for now, let's focus on what this paper actually achieves, because it's a significant step forward.

Conclusion: Tom: Well, we've covered a lot of ground on this paper — "mmWave Radar Aware Dual-Conditioned GAN for Speech Reconstruction of Signals With Low SNR." Let's wrap this up with a summary of what we've learned.

Jane: Absolutely, Tom. So the core achievement here is that the team from BITS Pilani built a system that can reconstruct intelligible speech from extremely noisy, band-limited radar signals. They did this by focusing on the reliable low-frequency content, using a two-stage training approach, and adding a clever fusion gate that combines raw radar data with an enhanced version from a separate module.

Tom: And the results are genuinely strong. They beat the previous state of the art, WaveVoiceNet, by a significant margin on their weighted score — zero point three three three versus zero point two six zero. And they did it with a training budget that's surprisingly modest — under a day on a single GPU.

Lu: I'd add that the implications go beyond just this specific task. This demonstrates that radar can be a viable sensing modality for speech, which opens up applications in search and rescue, security, and even assistive technology for people who can't use traditional microphones.

Meng: And from an engineering standpoint, the fact that it's a relatively small model with no pretrained components means it's reproducible and adaptable. That's a big deal for the research community.

Lalam: I'd like to add one more perspective. This technology has the potential to bridge communication gaps in ways we haven't fully explored. In disaster scenarios, in inaccessible environments, or even in everyday settings where microphones fail, this could give a voice to people who would otherwise be unheard. The cultural impact of that — being able to communicate in situations where we currently can't — is profound.

Jane: That's a beautiful way to put it, Lalam. And with that, I think we've given this paper the attention it deserves. We'll be back next time with another exciting piece of research. Until then, keep listening, keep learning, and keep asking questions.

Tom: Thanks for joining us, everyone. This is Tom and Jane, signing off.

Jash Karani, Adithya Chittem, Deepan Roy, Sandeep Joshi

BITS Pilani

cs.SD, cs.LG

Submitted: 2026-08-15

Updated: 2026-08-18

Comments: Under review at Interspeech 2026

Project page: https://rase-challenge.github.io/RASE2026-Challenge/#dataset

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 79/100

Key concepts

mmWave Radar
This technology, similar to adaptive cruise control sensors in cars, detects human speech by picking up tiny vibrations from the speaker's diaphragm or nearby objects. It can detect these vibrations through materials like glass walls.
GAN (Generative Adversarial Network)
A type of neural network used here where a generator tries to create realistic speech from radar data, and a discriminator tries to distinguish the generated speech from real clean speech. They compete until the generator produces convincing results.
Dual-Conditioned GAN
This specific GAN uses two different inputs during training: clean, low-frequency speech used for initial reconstruction, and noisy radar data for fine-tuning. This dual conditioning helps the model learn to handle the extreme noise of the radar signal.
Fusion Gate
A module that intelligently combines two inputs—the raw noisy radar data and an enhanced version from another module. It decides frame by frame which input to trust more based on its perceived reliability.

Terminology

Summary

Summary

This paper proposes a two-stage speech reconstruction pipeline for millimeter-wave (mmWave) radar using a Radar-Aware Dual-conditioned Generative Adversarial Network (RAD-GAN), which is capable of performing bandwidth extension on signals with low signal-to-noise ratios (-5 dB to-1 dB), captured through glass walls. The authors state: We propose an mmWave-tailored Multi-Mel Discriminator (MMD) and a Residual Fusion Gate (RFG) to enhance the generator input to process multiple conditioning channels. The proposed two-stage pipeline involves pretraining the model on synthetically clipped clean speech and finetuning on fused mel spectrograms generated by the RFG. The authors empirically show that the proposed method, trained on a limited dataset, with no pre-trained modules, and no data augmentations, outperformed state-of-the-art approaches for this specific task.

The system model consists of six main components: a HiFi-GAN-based generator, a set of three adversarial discriminators, a WaveVoiceNet (WVN) module, and a fusion gate used during fine-tuning. The generator maps an 80-bin mel-spectrogram to a waveform through transposed-convolution upsampling with Multi-Receptive Field (MRF) residual fusion blocks, conditioned only on the mel input (no stochastic noise input). The discriminators include the standard HiFi-GAN waveform discriminators (Multi-Period Discriminator and Multi-Scale Discriminator) plus the proposed Multi-Mel Discriminator (MMD), which consists of two parallel 2D convolutional discriminators applied to mel-spectrogram inputs of shape (B, 1, nmels, T) with one branch using spectral normalization and the other using weight normalization. The MMD produces a patch level score map rather than a single scalar, enabling the discriminator to assess local realism across different regions of the spectrogram.

The WaveVoiceNet module is adopted from prior work and used as an additional conditioning branch for HiFi-GAN during fine-tuning. The Residual Fusion Gate (RFG) fuses noisy and WVN conditioning using a residual gate, where Mn is the carry baseline, Mw − Mn is the WVN residual correction, Mf is the fused mel, G ∈ [0, 1]B×F ×T is a local mask, and σ(a) is a global correction scale. The gate uses a pointwise Conv1D (2F →F, i.e., 160→80), operating frame-wise and learning cross-frequency mixing across mel bins without temporal smoothing.

The loss functions include standard least-squares adversarial and feature-matching objectives from HiFi-GAN, plus an L1 mel loss with high-frequency weighting (λmel = 45.0, wm = 5.0 for mel bins above a cutoff frequency) and a multi-resolution short-time Fourier transform (MR-STFT) loss (λstft = 5.0). In Phase 1 (pre-training), we deliberately exclude all adversarial and feature-matching losses and train the generator using only spectral reconstruction objectives. In Phase 2 (fine-tuning), adversarial supervision is introduced... to refine perceptual quality.

The dataset was provided as part of the RASE 2026 Challenge, using a TI AWR2243BOOST mmWave FMCW radar capturing data through a glass wall. Task 1 (direct diaphragm vibration) contains 6,093 paired speech samples (5,334 train, 759 validation), and Task 2 (secondary surface vibration from aluminum foil) contains 5,978 paired samples (5,229 train, 749 validation). All speech samples are sampled at 8 kHz with an average duration of 6.4s, and the global SNR for radar-captured speech lies in the range of-5 dB to-1 dB.

Pre-training uses band-limited clean speech (to 1 kHz) with 80-dimensional Mel spectrograms (fmin = 0 Hz, fmax = 1000 Hz), optimized using MR-STFT loss with FFT sizes 256, 512, 1024, hop sizes N/4, and spectral weights (wsc, wlogmag, wlinmag) = (1.0, 1.0, 0.0). Training uses AdamW with β = (0.9, 0.99), initial learning rate 10−4, exponential learning rate scheduler with γ = 0.999, run for 66k steps with batch size 16 on an NVIDIA A6000 GPU (≈6h). Fine-tuning initializes from pre-training, uses the full Phase-2 objective, and runs for 100k steps with batch size 16 for ≈14h. The model has 87,000,536 trainable parameters.

Evaluation metrics include PESQ, ESTOI, DNSMOS, and MFCC Cosine Similarity. The authors define a weighted score combining normalized PESQ, DNSMOS, CSMFCC, and ESTOI, with Weighted score = 0.4 Task 1 + 0.6 Task 2 because Task 2 is the more challenging setting. RAD-GAN achieves the best overall weighted score (0.333) and best per-task scores (Task 1: 0.387, Task 2: 0.297), outperforming WVN (0.260) and HiFi-GAN (0.288). The authors note: RAD-GAN's gated residual fusion and two-stage training make better use of low-band cues for stable high-frequency recovery, which is effective in this low-data, low-SNR regime.

The ablation study shows incremental improvements: adding MMD+MR-STFT without pretraining gives a small gain (+0.002), pretraining provides a larger jump (+0.022), and WaveVoiceNet conditioning gives an additional +0.021, for a total improvement from 0.288 (baseline) to 0.333. Qualitative results show RAD-GAN reconstructs clearer upper-band harmonics, preserves the 2.6–3.2 s silence region with less leakage, and follows the clean waveform envelope more closely with sharper onsets/offsets and stronger peak capture.

The authors conclude: "Speech reconstruction from mmWave radar is difficult because the observations are low-SNR, band-limited, and information-poor (effectively a 1 kHz→4 kHz bandwidth extension setting). We addressed this with a two-stage pipeline: pretraining + fine-tuning with WVN-guided gated Mel fusion for robust conditioning. Future work will focus on real-time deployment by reporting latency, followed by model compression through distillation for edge inference."

Improvements for AI systems

Based on the paper, here are the specific improvements I can implement in AI systems:

  • Improvement: Add a two-branch 2D mel-spectrogram discriminator (spectral-norm + weight-norm) alongside existing waveform discriminators.

  • What it does: Provides complementary time-frequency supervision, improving spectral realism and training stability in low-SNR conditions. The patch-level score map allows local realism assessment across spectrogram regions.

  • Improvement: Implement a gated residual fusion mechanism: G = σ(Conv1×1([Mn; Mw−Mn])), Mf = Mn + σ(a)·G⊙(Mw−Mn), with gate bias and scale initialized to −2.0.

  • What it does: Dynamically blends noisy mel input with enhanced mel from a secondary model, allowing fallback to noisy cues when enhancement is unreliable, and amplifying enhancement in reliable regions. This improves robustness to varying noise levels.

  • Improvement: Phase 1: Train generator with only spectral reconstruction losses (L1 mel loss with high-frequency weighting + MR-STFT) on synthetically band-limited clean speech (0–1 kHz). Phase 2: Finetune with adversarial losses (MPD, MSD, MMD) + feature matching + reconstruction losses on real noisy radar data.

  • What it does: Isolates bandwidth extension learning from GAN instability, leading to more stable convergence and better final quality. The high-frequency weighting (wm=5.0 above cutoff) explicitly penalizes errors in the 1–4 kHz band.

  • Improvement: Use L mel = λmel·E[Σ wmφ(x)−φ(x̂)] with wm=5.0 for mel bins above cutoff frequency, wm=1 below, λmel=45.0.

  • What it does: Forces the model to prioritize accurate reconstruction of upper frequency bands, which is critical for intelligibility in bandwidth extension tasks.

  • Improvement: Add MR-STFT loss with FFT sizes 256, 512, 1024, hop sizes N/4, spectral weights (wsc=1.0, wlogmag=1.0, wlinmag=0.0), λstft=5.0.

  • What it does: Captures both fine temporal detail and long-range structure, improving spectral fidelity and reducing artifacts.

  • Improvement: Initialize RFG gate bias and learnable scale a to −2.0, making the gate start in a pass-through mode (σ(−2)≈0.12).

  • What it does: Prevents the fusion gate from making aggressive corrections early in training, reducing risk of destabilizing the generator when the conditioning model is still unreliable.

  • Improvement: Condition the generator on 80-bin mel spectrograms with fmin=0, fmax=1000 Hz, but compute all losses over the full 0–4 kHz bandwidth of target and generated waveforms.

  • What it does: Explicitly forces the model to learn plausible high-frequency structure from low-frequency cues, rather than memorizing noisy high-band artifacts.

  • Improvement: Use weighted score = 0.4·Task1 + 0.6·Task2, where each task score combines normalized PESQ, DNSMOS, MFCC cosine similarity, and ESTOI.

  • What it does: Provides a single, balanced metric that rewards models performing well on the harder task (Task 2: secondary surface vibration), enabling fair comparison across approaches.

Capabilities of the improved AI system:

  • Reconstructs intelligible full-bandwidth speech (0–4 kHz) from severely degraded, band-limited (0–1 kHz) inputs with SNR as low as −5 dB

  • Works with limited training data (6 hours) without pretrained models or data augmentation

  • Maintains performance across both direct and indirect vibration capture scenarios

  • Produces perceptually natural speech (DNSMOS 2.688 vs 2.286 baseline) with better intelligibility (ESTOI 0.190 vs 0.144 baseline)

  • Stable training convergence through two-stage optimization, avoiding GAN collapse in low-SNR regimes

Sources

Related papers