Learning-Based Behavior Planning for Automated Driving: Real-World Integration and Deployment

This paper proposes a hybrid planning architecture for automated driving that combines learning-based behavior planning with optimization-based trajectory supervision, aiming to exploit the generalization capabilities of learned behavior generation while retaining deterministic, constraint-based safeguarding of the resulting trajectories. The architecture is designed in a modular manner with clearly separated responsibilities, allowing the learning-based behavior planner to be exchanged and updated without modifying the safety-critical supervision and fallback modules. This separation also facilitates an MLOps workflow in which operational data can be collected, incorporated into subsequent training and validation cycles, and used for the continuous refinement of the behavior planner.

The trajectory supervision module safeguards the learning-based reference trajectory before it is passed to downstream modules. Its role in the architecture is to transform the reference provided by the behavior planning module into a drivable trajectory while enforcing explicit safety, traffic-rule compliance, drivability, and vehicle-dynamic constraints. For this purpose, a trajectory optimization framework is developed that formulates and solves an optimal control problem (OCP) using a kinematic single-track model, with constraints on states, controls, obstacle avoidance, drivable space, and vehicle-dynamic feasibility. The optimization framework is built on acados, which is used to formulate the nonlinear OCP and generate solver libraries for online execution within a ROS 2 node.

In addition to the learning-based planning path, the proposed architecture includes a deterministic safety fallback. Its role is to provide a reliable, rule-based planning branch that continuously produces a drivable trajectory. If a regular fallback trajectory cannot be generated due to invalid inputs, unavailable information, or unknown traffic situations, the module initializes a safe-stop trajectory. The fallback is therefore a complementary safety layer for degraded operation and is not intended to replace the learned behavior planning during nominal operation. For the fallback path, a simple centerline-based planner is developed, which constructs a reference trajectory along a given map-based centerline, including encoded speed limits as reference velocity, and adapts longitudinal behavior based on specified traffic situations.

The module that determines which trajectory is to be tracked by the downstream trajectory controller is represented by a switch. In the presented setup, the supervised output of the learning-based planner is forwarded to the trajectory controller by default. If the reference trajectory is of insufficient quality and the supervision module is therefore unable to ensure drivability and safety, the switch forwards the output of the deterministic fallback path. In these cases, it is also possible to collect the necessary data for generating new training sets so that the behavior planning can handle these scenarios in the future as well.

The trajectory controller is the final module of the presented architecture and converts the selected drivable trajectory into actuator commands for vehicle execution. It is a cascaded proportional-integral-derivative (PID)-based Ackermann trajectory controller, structured into a longitudinal and a lateral control path, both combining feed-forward terms from drivable trajectories with feedback PID control. PID gains are scheduled over the operating velocity range, allowing the controller behavior to be adapted to varying vehicle dynamics. To improve overall robustness, the planning architecture follows a bi-level stabilization concept: during regular operation, trajectory supervision uses the interpolated state from the previously generated trajectory as the initial state for the next optimization, and whenever predefined thresholds in longitudinal or lateral tracking are exceeded, replanning from the actual vehicle state is initiated.

The behavior planning module is implemented as a deep neural network (DNN) in PyTorch and predicts a reference trajectory based on a policy learned through supervised learning from selected data. The neural network does not operate on raw sensor data, but on the processed output of the upstream environment model, including HD map, navigation, tracked object list, and ego state information. The network architecture builds on a vectorized scene representation and adopts design principles from recent imitation-based transformer planners, including attention-based scene fusion mechanisms. The input scene representation is described in an ego-centric bird’s-eye view, where all elements are expressed relative to the current ego pose. Each agent is represented by a short state history, including position, heading, velocity, and bounding box size over the past 2 seconds. The vectorized map comprises lane centerlines with their width and type, lane boundaries with their type and whether they may be crossed, and regulatory elements such as traffic lights, yield lines and right-of-way rules together with their current state.

The planner primarily outputs a reference trajectory over an 8 s horizon at 0.5 s spacing, represented as a sequence of positions and velocities. Rather than regressing independent waypoints, the decoder builds the trajectory stepwise, predicting a velocity and a heading change at each step and integrating them using a simple kinematic model. Additionally, two auxiliary tasks are learned jointly with the trajectory: a short-horizon prediction of the surrounding objects, produced by a separate decoder, and a set of map-grounding quantities read from the ego decoder’s per-step state, namely the lateral offset to the route and the distance to the drivable corridor. The attention weights of the fusion and the decoder remain accessible as an interpretability output.

The planner is trained by imitation, where for each recorded scene, the network is trained to predict the corresponding expert trajectory, typically originating from human driving behavior. The loss combines four imitation terms (position, velocity, heading, and heading change) with an auxiliary multi-task term. The velocity and heading-change terms are weighted more strongly than position and absolute heading, since a faithful velocity profile and turn-in behavior are key to reproducing the reference trajectory.

Three models are trained on different datasets. The DrivIng dataset serves as the primary data source for training, comprising three recordings of the same route, approximately 18 km long, captured at three different times (day, dusk, and night) with a total duration of approximately 105 minutes. All three models are trained on the day and night recordings of the DrivIng dataset, while the dusk recording was held out for validation, resulting in 34,438 training samples and 16,518 validation samples. To study transferability to a different domain, two additional datasets were recorded on ika’s test track using the research vehicle karl.: one capturing driving without dynamic objects and one including dedicated interaction scenarios with an additional vehicle. The first model is trained exclusively on DrivIng, while the second and third models additionally incorporate the test track and test track interaction datasets, respectively.

On the DrivIng validation split, the errors remain small: over the 8 s horizon the position ADE is 1.83 m, the velocity ADE 0.56 m/s, and the heading ADE 2.6°. Across the entire DrivIng validation set, the model never proposes to cross a red light, and 2.3 % of the predicted 8 s trajectories would have led to a collision, assuming that the ego vehicle followed the predicted trajectory and all other agents followed their recorded future trajectories. Restricting the evaluation to the first 4 s provides a more meaningful assessment in practice, with a lower corresponding collision rate of 0.3 %. Counterfactual validation examples show that changing the traffic light state from green to red causes the predicted trajectory to stop before the stop line, and inserting a stopped lead vehicle causes the planner to slow down behind the object.

Transfer to a different domain is assessed on the test track recordings, whose road geometry is unseen by a model trained exclusively on DrivIng. Zero-shot, performance is clearly degraded because the training data does not sufficiently cover the new road geometry, which shows in the much larger heading and velocity errors, and the model collides frequently in the interaction scenarios (26.7 % at 8 s). When additionally trained on test track data, however, the model is able to follow the test track geometry, while the interaction behavior learned from DrivIng transfers effectively to the new domain. The resulting model remains collision-free over the 4 s horizon, with only a few number of residual collisions occurring over 8 s. A small number of test track interaction samples removes these residual collisions as well. The auxiliary tasks contribute to this transfer: trained without them, the same +test track model still collides in 20.6 % of the 8 s interaction trajectories, which the auxiliary objectives reduce to the 5.9 %. Conversely, incorporating the limited set of test track interaction examples does not degrade performance on DrivIng; instead, it yields a slight improvement, reducing the collision rate on the validation set from 2.3 % to 1.2 %.

The proposed hybrid planning architecture was deployed on the research vehicle karl. on ika’s test track. A route was defined that covers lane following, lane changing, turning, and interaction with another road user. During the conducted test runs, the behavior planning module generated trajectory predictions at a frequency of 10 Hz and provided references that followed the predefined route and did not cause collisions. Qualitatively, the snapshots indicate that the supervised trajectory largely preserves the maneuver represented by the learning-based reference while converting it into a feasible trajectory for vehicle execution. In comparison, the safety fallback exhibits more conservative behavior, remaining closer to the lane center and decelerating more strongly when approaching the interaction area. The learning-based reference, by contrast, follows a smoother and less centerline-bound path through the curve, which is largely retained by the trajectory supervision. The test track demonstration serves as a proof of concept for the real-world integration of the proposed hybrid planning architecture, providing qualitative evidence that the individual modules can operate jointly on the research vehicle under real-world sensing, actuation, and timing conditions. Additionally, the integrated system was demonstrated in real-vehicle operation at the autotech.agil final event at the Aldenhoven Testing Center, providing additional qualitative evidence that the full stack can operate stably in a real-world demonstrator setting.

The paper concludes that the results highlight the potential of hybrid architectures to couple data-driven performance with deterministic supervision and safeguarding, while additionally enabling continuous, data-driven improvement of the driving behavior. Future work will focus on a broader quantitative evaluation of the integrated system in more diverse and complex traffic scenarios, including evaluating the planning architecture in public road traffic once the required approval is granted.

Improvements to AI Systems:

1. Hybrid Data-Driven + Constraint-Based Planning Architecture: Integrate a learning-based trajectory generator with a deterministic optimization-based supervisor that enforces safety, traffic rules, drivability, and vehicle dynamics. This allows AI to generalize to novel scenarios while guaranteeing constraint satisfaction before execution.

2. Modular MLOps-Enabled Design: Separate the learned planner from safety-critical supervision and fallback modules, enabling continuous retraining on operational data without re-validating the entire stack. The system can automatically collect data when the fallback is activated, then use it to retrain the planner to handle previously unseen edge cases.

3. Kinematic Single-Track Trajectory Optimization: Use an optimal control problem (OCP) solver (e.g., acados) to transform raw learned references into dynamically feasible trajectories, with explicit constraints on obstacle avoidance, drivable space, and vehicle dynamics. This improves real-world executability and reduces reliance on post-hoc filtering.

4. Deterministic Rule-Based Fallback with Safe-Stop: Implement a centerline-based planner that continuously generates a drivable trajectory and, if inputs are invalid or unknown, initializes a safe-stop trajectory. This ensures graceful degradation and prevents catastrophic failures in degraded operation.

5. Bi-Level Stabilization with Replanning Triggers: Use interpolated states from previous trajectories as initial conditions for optimization, and trigger replanning from the actual vehicle state when longitudinal or lateral tracking errors exceed thresholds. This improves robustness to disturbances and model mismatch.

6. Stepwise Kinematic Trajectory Decoding: Instead of regressing independent waypoints, decode trajectories by predicting velocity and heading changes stepwise and integrating them via a kinematic model. This produces smoother, more physically consistent paths and improves turn-in behavior.

7. Auxiliary Multi-Task Learning for Transfer: Jointly learn short-horizon object prediction and map-grounding quantities (lateral offset, distance to drivable corridor) alongside trajectory prediction. This improves zero-shot transfer to new domains and reduces collision rates significantly (e.g., from 20.6% to 5.9% in interaction scenarios).

8. Attention-Based Scene Fusion with Interpretability: Use transformer-style attention to fuse vectorized map, object histories, and ego state, while exposing attention weights as interpretability outputs. This enables debugging and trust in the planner’s decisions.

9. Counterfactual Validation for Safety: Train and test with counterfactual scenarios (e.g., changing traffic light state, inserting stopped vehicles) to verify the planner’s rule compliance and reactive behavior, ensuring it stops at red lights and slows for obstacles.

10. Domain-Adaptive Training with Mixed Datasets: Train on diverse datasets (e.g., public roads + test track) to improve generalization, and add small amounts of domain-specific interaction data to eliminate residual collisions while preserving performance on the original domain.

What the Improved AI System Can Do:

• Operate autonomously in real-world traffic with a learned planner that handles complex maneuvers (lane changes, turns, interactions) while a supervisor guarantees safety and feasibility.

• Continuously improve by collecting data from fallback activations and retraining the planner, reducing future fallback reliance.

• Transfer to new environments (e.g., test tracks, unseen road geometries) with minimal fine-tuning, thanks to auxiliary tasks and mixed training.

• Explain its decisions via attention weights, aiding human oversight and regulatory compliance.

• Degrade gracefully in sensor failures or unknown situations by switching to a deterministic safe-stop trajectory.

• Maintain stable control under real-world timing and actuation constraints, as demonstrated on a research vehicle, with replanning triggered by tracking errors.

Recent research in machine and deep learning has shown the potential of learningbased motion planning approaches to improve the driving behavior of automated vehicles, especially in complex environments. However, their complex nature and lack of transparency can hinder explainability and trustworthiness and complicate safety assurance. Motivated by these challenges, we propose a hybrid planning architecture that combines the advantages of machine learning with the verifiability and the determinism of classical approaches. Specifically, we developed a deep neural network to interpret complex traffic scenes and propose driving behavior, while an optimization-based supervision layer validates this proposal and enforces explicit drivability and safety constraints. We evaluate the learned planner's driving behavior in open-loop studies on real-world urban data, discuss system integration aspects for stable closed-loop operation, and report results from real-world deployment on our research vehicle karl..