Drive As You Like: Multi-Head Diffusion with Reinforcement Learning for Personalized Driving

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "Drive As You Like".

Dev: The gist The proposed M-Diffusion Planner is an RL-guided multi-strategy framework with a diffusion-based multi-head planner integrated with LLM-based semantic understanding,

Rosa: First, who's behind it and why it matters.

Paper summary: Rosa: We're diving into the specifics of "Drive As You Like: Multi-Head Diffusion with Reinforcement Learning for Personalized Driving." This paper argues that existing imitation learning systems are too biased toward common driving patterns, missing the diversity of human driving behaviors, and they struggle to understand what a user actually wants from them.

Dev: They propose an M-Diffusion Planner as the solution. The thesis is that we need a framework that combines LLM-based semantic understanding with a diffusion-based multi-head planner to enable dynamic perception of user intent and generate trajectories aligned with those diverse needs.

Taro: So, in plain terms, the system starts by using an LLM to interpret what the user is asking for in their interaction. This interpretation then selects one specific driving strategy from a set of available strategies that are all controlled by different heads within a shared diffusion planner.

Rosa: That's right. The key feature they highlight is this dual decoupling: separating the intent understanding part from the trajectory planning part, and learning the general planning capability across those different strategies through strategy alignment.

Dev: They also introduce a lightweight GRPO-based strategy alignment method, which freezes the shared backbone of the planner and only optimizes those specific output layers with a relative preference reward. This allows for efficient adaptation without needing huge amounts of new data for every single driving style.

Taro: It sounds like they are trying to solve the problem where you can't just have one perfect policy that covers every possible driving scenario because human needs change constantly.

Rosa: Precisely. They show that after pre-training with imitation learning, these strategy-specific rewards specialize each head to match user preferences, which is what allows for those diverse driving styles without requiring explicit style annotations in the initial data.

Dev: The architecture itself is quite complex on the input side. They have an Encoder using an MLP-Mixer followed by a Transformer layer that encodes mixed scenario inputs into compact embeddings.

Taro: And then the Decoder is built on a DiT, which allows for trajectory generation conditioned not just on the general context but also directly on those high-level planning strategies selected by the LLM.

Rosa: The multi-level fusion part is also important because they concatenate the future trajectory with current vehicle states to give it a clear starting point, and they fuse historical features using an MLP-Mixer that mixes tokens along both temporal and spatial dimensions.

Dev: They also encode static objects like road boundaries and signs using an MLP and Transformer block incorporated through multi-head cross-attention, as well as injecting global semantic cues into the diffusion process via a temporal condition.

Taro: So, it’s a lot of conditioning going into the process—contextual encodings, historical features, object encodings, and global navigation cues—all feeding into that final output layer selection.

Rosa: That's the core of M-Diffusion Planner: taking rich scenario inputs and high-level strategy instructions to generate highly personalized motion plans. This framework aims to bridge the gap between simple behavior regression and truly context-aware, user-dependent driving requirements.

Conclusion: Dev: So looking at "Drive As You Like: Multi-Head Diffusion with Reinforcement Learning for Personalized Driving," the authors Ding Fan et al. are proposing this M-Diffusion Planner as a novel way to handle personalized driving needs through RL and LLMs.

Rosa: The title itself, "Drive As You Like," suggests a system that is finally capable of adapting its behavior to what the user likes or wants in a driving situation, not just rigidly following pre-programmed rules.

Taro: What this means practically is that instead of being stuck reproducing high-frequency behaviors from old datasets, an autonomous vehicle could potentially switch between different driving modes on demand based on the driver's current needs.

Dev: They are showing that you can use a diffusion model structure to generate diverse, high-quality trajectories and then use RL to fine-tune specific heads for each strategy. This allows for real-time strategy switching without needing a full model retraining or reloading every time the user changes their mind.

Rosa: The conclusion is that this framework offers both adaptability and efficiency for interactive driving applications, allowing the planner to support trajectory generation conditioned on multi-modal user instructions.

Taro: It suggests a future where autonomous vehicles are not just predictable machines but truly responsive partners that can adjust their style based on the user's specific context.

Dev: The paper demonstrates that this approach delivers competitive or superior planning performance when evaluated on benchmarks like nuPlan, showing distinct strategy-aligned behaviors.

Rosa: So, in short, M-Diffusion Planner is a framework that leverages diffusion models and reinforcement learning to create a system that can generate varied, high-quality driving plans tailored to individual user requirements dynamically.