Threat-guided Policy-aware Scene Perturbation for Safe Autonomous Driving with Online Reinforcement Learning

TPSP introduces a policy-aware scene encoder to capture the interaction between policy behaviors and surrounding environments, enabling scene perturbation aligned with the current policy. Based on this representation, TPSP selectively perturbs critical objects rather than applying uniform modifications across the scene. Furthermore, we develop a threat-guided optimization strategy that evaluates perturbed scenes through threat-level differences between policy rollouts on original and perturbed scenes, guiding the generation of safety-critical scenes with higher training value. Comprehensive experiments demonstrate that TPSP improves safety learning efficiency, achieving strong safety performance on NAVSIM v2 with approximately 4 million kilometers of simulated driving data. Ablation studies verify that policy-aware targeted perturbations provide more informative safety-critical experiences than random or policy-unaware strategies, enabling safer driving under limited interaction budgets.

TPSP consists of three key components: a policy-aware scene encoder, a targeted scene perturbation module, and a threat-guided scene perturbation optimization method. First, the policy-aware scene encoder integrates ego states, surrounding objects, road context, scene risk information, and the current policy’s action distribution to construct a policy-aware scene representation. This representation provides both scene-level and policy-level information for perturbation generation. With this representation, targeted scene perturbation selectively modifies critical objects instead of applying uniform perturbations across the scene. Finally, TPSP evaluates perturbed scenes by computing the threat-level differences between the policy rollouts on perturbed and original scenes. The resulting threat signal optimizes the scene perturbation network, guiding it toward discovering safety-critical scenes with higher training value for autonomous driving policy improvement.

On the NAVSIM v2 navhard two stage benchmark (Cao et al. 2025), TPSP achieves 99.8% NC and 99.6% TTC on Stage 1 and maintains strong performance in the more challenging Stage 2 setting with 96.7% NC and 94.6% TTC, outperforming previous best results by 2.2% and 1.8%, respectively.

The contributions are summarized as follows:

• We propose TPSP, a novel threat-guided perturbation framework designed to generalize safety-critical scenes for safe autonomous driving. By targeting the current policy, TPSP generates high-risk interaction cases to robustly train safe driving behaviors.

• Within TPSP, we quantify the threat by comparing pre- and post-perturbation rollouts to identify high-risk samples. The scene perturbation module is then trained with the objective of maximizing these threat scores.

• Extensive experiments demonstrate that TPSP significantly outperforms baseline methods in both driving safety and data efficiency, highlighting its effectiveness in training robust driving policies.

Existing methods typically characterize scene difficulty based on environment-level objectives, such as collision probability, adversarial objectives, or predefined safety violations. Consequently, the generated scenes are often independent of the evolving policy: a specific event, such as a cut-in or sudden braking, may be trivial for a mature policy but challenging for an immature one. This limitation prevents existing difficulty-oriented strategies from effectively targeting the current policy weaknesses during online RL training.

The policy-aware scene encoder extracts heterogeneous features from the scene, including ego feature, surrounding object features, road feature, risk-related feature and policy feature conditioned on st. The ego feature eego encodes the current kinematic state of the ego vehicle, including velocity, heading, and destination information. For the i-th surrounding object, its feature representation eobj i contains intrinsic attributes, such as object type and geometric properties, as well as interaction-related information with the ego vehicle, including relative position, relative velocity, and Time-to-Collision (TTC). The road feature eroad encodes the information of the surrounding road structure. TPSP incorporates a scene-level risk-related feature erisk that aggregates several safety-related indicators, including the inverse minimum distance to surrounding objects, the inverse minimum TTC, and the number of surrounding objects in the scene. To capture the characteristics of the current driving policy, TPSP extracts a policy feature from the hidden representation of the frozen policy network, obtained through a projection network: epolicy = MLPpolicy(zπθ̄).

The complete policy-aware scene representation is obtained by fusing all feature components: h = MLPfusion([eego, cobj, eroad, erisk, epolicy]), where cobj denotes the aggregated object interaction features from an attention module.

For targeted scene perturbation, TPSP assigns each observable object an importance score using a Softmax function: wobj = Softmax(f score([eobj i, h])), and selects the top-K objects as perturbation targets. The perturbation network Gϕ models a Gaussian perturbation distribution and samples raw perturbation variables: z raw ∼ N(µ, diag(σ2)). The final perturbation variables z correspond to three types of scene modifications: object speed adjustment, object spawn time shift, and ego initial state modification.

For threat-guided optimization, TPSP computes a step-level threat score: ct = wttc·cttc t + wdist·cdist t + wedge·cedge t + wcol·ccol t + woff·coff t, representing time-to-collision, inter-object distance, road-edge proximity, collision occurrence, and off-road status. The rollout-level criticality is computed as C(τ) = (1/k′)Σ t∈Top(τ,k) ct, selecting the top-k timesteps with the highest step-level threat scores. TPSP also introduces the threat instability metric V(τ) = (1/T)Σ t=0 T-1ct+1 − ct. The final threat score is ξ(τ) = C(τ)·(1 + βV(τ))/(1 + β).

The threat difference between perturbed and original rollouts is ∆ξ = ξ(τ per) − ξ(τ org), generated using the same fixed driving policy πθ̄ and identical simulator seeds. The perturbation network is optimized with: Lper = −Es[Âs log pϕ(z raw sE obj K, h)] − cent·Hz ent, where Âs is the normalized threat advantage.

In the safety learning efficiency analysis, TPSP achieves faster improvements in both collision avoidance and off-road safety rewards compared with Vanilla PPO. At 2M miles, TPSP already reaches a collision-reward level that Vanilla PPO does not attain until 4M miles.

In the ablation study comparing Vanilla PPO, random perturbation, TPSP without policy awareness (TPSP w/o PA), and the full TPSP framework, TPSP achieves the best performance across both stages, particularly in the more challenging S2 setting, where it obtains 96.7% NC and 94.6% TTC. Random perturbation provides only limited improvements over Vanilla PPO, indicating that increasing scene diversity alone is insufficient for effective safety learning. The performance gap between TPSP and TPSP w/o PA demonstrates that incorporating policy-specific information is essential for generating more informative safety-critical scenes.

In the qualitative analysis, in a cut-in situation, the generated perturbation increases the threat of the interaction with ∆ξ = +0.1115, corresponding to a more dangerous merging conflict. In contrast, a car-following situation exhibits a less hazardous interaction pattern, and the estimated threat variation ∆ξ = −0.0540 is consistent with this intuitive observation.

TPSP differs from existing NAVSIM methods in both learning paradigm and system pipeline. Existing approaches mainly improve driving performance by developing stronger perception and planning models from offline data, whereas TPSP focuses on improving the quality of online policy training experiences through simulator-based interaction. TPSP utilizes structured white-box information available from the simulator, including ego states, surrounding object states, road information, and risk-related features.

Improvements to AI systems:

1. Policy-aware scene perturbation for targeted safety training: Instead of generating random or uniformly modified training scenarios, the AI system can now use a policy-aware scene encoder that fuses ego states, object interactions, road context, risk metrics, and the current policy’s hidden representation. This allows the system to selectively perturb only the top-K critical objects (e.g., adjusting speed, spawn time, or ego initial state) that are most likely to expose the current policy’s weaknesses, rather than wasting interaction budget on irrelevant scene changes.

2. Threat-guided optimization for adaptive curriculum generation: The system can now compute a rollout-level threat score that combines time-to-collision, inter-object distance, road-edge proximity, collision occurrence, and off-road status, along with a threat instability metric to capture sudden risk changes. By maximizing the threat difference between perturbed and original rollouts (using the same frozen policy and simulator seeds), the AI system can automatically generate safety-critical scenes that are specifically challenging for the current policy version, enabling more efficient online RL training.

3. Data-efficient safety learning with limited interaction budgets: The improved AI system can achieve faster convergence in collision avoidance and off-road safety rewards—reaching at 2M miles what vanilla PPO requires 4M miles to achieve. This means the system can train safer autonomous driving policies with roughly half the simulated driving data, reducing computational cost while improving robustness.

4. Policy-adaptive difficulty calibration: Unlike environment-level difficulty metrics (e.g., collision probability or predefined violations), the system can now dynamically adjust scene difficulty based on the policy’s maturity. A scene that is trivial for a mature policy but challenging for an immature one is correctly identified and used only when it provides training value, preventing wasted episodes on either too-easy or too-hard scenarios.

What the improved AI system can do:

• Autonomously generate high-risk, policy-specific driving scenarios (e.g., dangerous cut-ins, sudden braking, merging conflicts) that target the exact weaknesses of the current driving policy during online RL training, leading to stronger safety performance (e.g., 96.7% NC and 94.6% TTC on challenging benchmarks) compared to random or policy-unaware methods.

• Optimize its own training curriculum in real time by evaluating threat-level differences between original and perturbed rollouts, ensuring that every simulated interaction provides maximal safety-learning value, even with limited interaction budgets.

• Achieve superior safety metrics with higher data efficiency, enabling deployment in environments where collecting large-scale real-world or simulated driving data is expensive or time-constrained.

• Adapt to policy evolution—as the policy improves, the system automatically shifts perturbation targets to new weaknesses, maintaining a continuous and effective safety training loop without manual scene design or predefined difficulty thresholds.