Vault: One-Step Latent Generation with Positive-Anchored Rewards for Autonomous Driving

One-step latent generation with positive-anchored rewards for autonomous driving addresses the latency constraints of iterative planning models by replacing multi-step denoising chains with a single forward pass conditioned on learned semantic features. The gist: A novel framework achieves efficient, multi-modal trajectory generation by employing a single-step conditional drift in a latent space guided by positive anchors and reward signals.

How it works

The core of the method involves replacing the iterative denoising process common in diffusion models with one-step latent generation. This is achieved through a conditional drift mechanism within a learned latent space, which allows for the production of diverse trajectory candidates in a single forward pass. The framework leverages this technique to bypass the unacceptable latency associated with multi-step sampling required by traditional diffusion planners.

The process utilizes several key components:

1. A visual encoder that processes sensor inputs to extract relevant scene information.

2. A VAE latent space where the generation occurs via conditional drift, controlled by a conditioning mechanism.

3. Positive anchors, which are used to guide the drift towards geometrically feasible and desirable trajectory regions in the latent space.

The Role of Positive Anchors and Rewards

Positive anchors are crucial for balancing diversity and precision during generation. The method constructs a drift target for each candidate that combines an attractive component and a repulsive component. The attractive component interpolates between two key points:

(1−α) · Vpos(i) + α · VGT

where Vpos(i) is the encoding of the positive sample matched to candidate i, and VGT is the expert ground truth. This mechanism ensures that candidates are attracted toward desirable behaviors while maintaining a degree of exploration.

The repulsive component is computed from other generated candidates' latent encodings:

(1/N − 1) · Xj̸=i VAEenc(τ j)

This combination of attraction and repulsion drives the drift target, which in turn dictates the trajectory generation process. Furthermore, a Winner-Take-All loss is applied to enforce precision by applying gradients only to the candidate closest to the ground truth:

(α · min i τ i − τGT1)

Semantic Guidance and Latent Generation

The framework integrates semantic reasoning into the latent generation process. The system uses learned features, such as those extracted from a visual encoder, to inform the generation strategy. This semantic guidance is essential for ensuring that the generated trajectories are not only diverse but also contextually appropriate for the given scene. The single-step drift is conditioned on these learned features, allowing the model to adapt its generation dynamics based on scene complexity.

Training and Evaluation

The training pipeline is designed to optimize both trajectory quality and generation efficiency. The system utilizes a reward signal derived from positive anchors and ground truth information during training to supervise the latent space exploration. This supervision guides the model in learning how to effectively utilize the single-step drift mechanism to produce high-quality outputs quickly. Evaluation involves assessing performance metrics on driving benchmarks, focusing on achieving high fidelity in trajectory prediction while maintaining real-time inference speeds enabled by the one-step generation process. The success of this approach is demonstrated by its ability to generate physically plausible and multi-modal trajectories efficiently.

Key Contributions

The primary contributions of this work lie in the combination of three key innovations:

1. Replacing iterative denoising with a single-step conditional drift in a VAE latent space for ultra-fast generation.

2. Incorporating positive-anchored rewards to guide the latent trajectory distribution toward expert and desired behaviors, balancing diversity and precision.

3. Developing an efficient mechanism that allows for high-fidelity, multi-modal planning without requiring dense geometric annotations or costly iterative sampling processes, achieving this in a single forward pass.

The gist

A novel framework achieves efficient, multi-modal trajectory generation by employing a single-step conditional drift in a latent space guided by positive anchors and reward signals.

The process utilizes several key components:

1 A visual encoder that processes sensor inputs to extract relevant scene information.

2 A VAE latent space where the generation occurs via conditional drift, controlled by a conditioning mechanism.

3 Positive anchors, which are used to guide the drift towards geometrically feasible and desirable trajectory regions in the latent space.

Positive anchors are crucial for balancing diversity and precision during generation. The method constructs a drift target for each candidate that combines an attractive component and a repulsive component.

Based on the CLEAR (Cognition and Latent Evaluation for Adaptive Routing) framework described in the paper, here are specific improvements for existing autonomous driving AI systems:

The proposed CLEAR framework offers several high-impact improvements over current state-of-the-art end-to-end planning methods (like diffusion models or deterministic regressors).

1. Improve Real-Time Latency and Safety Compliance

2. Enhance Multi-Modality Handling in Complex Scenarios

3. Reduce Reliance on Expensive Geometric Annotations

4. Integrate High-Level Cognitive Reasoning into Low-Level Control

Specific Capabilities of the Improved AI System:

1. Improve Real-Time Latency and Safety Compliance

This system can generate a diverse set of physically plausible trajectory candidates at extremely high frame rates (up to 99 FPS) through a single, efficient forward pass (single-step conditional drift in a VAE latent space). By replacing multi-step diffusion denoising chains with this method, the system meets stringent sub-100ms control budgets required for safety-critical deployment.

2. Enhance Multi-Modality Handling in Complex Scenarios

The system can effectively capture and generate diverse maneuvers (e.g., yielding, proceeding, merging) simultaneously by balancing two factors:

— It generates a wide variety of candidates by controlling the conditioning coefficient α (diversity).

— It ensures kinematic feasibility and expert precision through PCA projection and the Winner-Take-All loss mechanism (precision).

This allows the AI to handle inherently multi-modal situations, such as crowded unsignalized intersections, where deterministic methods fail by averaging incompatible maneuvers.

3. Reduce Reliance on Expensive Geometric Annotations

The system achieves high performance without requiring dense 3D perception annotations or exhaustive geometric reconstruction. It leverages:

— A frozen Drive-JEPA backbone to extract abstract geometric features from visual inputs.

— A compact, frozen VAE encoder for semantic structure in the latent space.

This means the AI can be deployed using only standard visual features and learned semantic representations, drastically reducing the need for costly, manual 3D mapping data.

4. Integrate High-Level Cognitive Reasoning into Low-Level Control

The system moves beyond purely geometric planning by incorporating a Large Language Model (Qwen 3.5 0.8B) as a semantic feature extractor:

— The LLM's hidden states encode traffic logic, interaction norms, and risk priors from driving QA pairs.

— These states drive an Adaptive Scheduler that dynamically selects the optimal sampling strategy (the correct α and N) for the current scene complexity.

The Cross-Attention Scorer then evaluates generated candidates against these learned cognitive semantics instead of relying solely on heuristic cost functions, leading to safer and more context-aware trajectory selection.