HENet++: Hybrid Encoding and Multi-task Learning for 3D Perception and End-to-end Autonomous Driving

The gist: HENet++ achieves state-of-the-art end-to-end multi-task perception performance on the nuScenes dataset while attaining the lowest collision rate on the nuScenes end-to-end autonomous driving benchmark.

Hybrid Encoding and Multi-task Learning

The framework introduces a hybrid image encoding network that uses a large image encoder for short-term frames and a small one for long-term frames to address computational constraints The proposed architecture maintains compatibility with various existing 3D feature extraction methods and supports multimodal inputs. Furthermore, the framework simultaneously extracts both dense and sparse features, providing more suitable representations for different tasks, reducing cumulative errors, and delivering more comprehensive information to the planning module.

Hybrid Image Encoding Network

The Hybrid Encoding Network utilizes two image encoders of distinct complexities to process different temporal inputs. The first encoder handles high-resolution shortterm frames, passing them through a large image backbone(e.g., VoVNetV2-99 [62]) and a feature pyramid network (FPN) [63], and subsequently applies a complex 2D-to-BEV network to generate high-precision BEV features. The second encoder processes longterm sequences by down-sampling the inputs to low resolution, using a lightweight backbone(e.g., ResNet-50 [64]) with FPN for efficient feature extraction, followed by a simplified 2D-to-BEV module. Both pathways employ BEVPoolv2 [65] to project frustum Algorithm 1 Pseudo-code for Section 3.2.

Temporal Feature Integration

Following the extraction of multi-frame BEV features by the hybrid image encoding network, they are fused using a temporal integration module, which operates through complementary backward and forward processes. This module proposes the adjacent frame fusion module (AFFM) and adopts the temporal fusion strategy with temporal backward and forward processes. The AFFM operation can be formulated as AFFM(fi, fj) =fj + γ × Avg(Atn(⟨fi, fj⟩, fi, fi), Atn(⟨fi, fj⟩, fj, fj)).

Independent BEV Feature Encoding

To mitigate conflicts between tasks and leverage task preferences for different feature sizes, HENet proposes a preliminary solution that provides separate BEV feature maps for different tasks. This involves assigning different-sized BEV features to other tasks after obtaining the fused multi-scale BEV features. Inspired by BEVFusion [67], the proposed encoding process comprises adaptive feature selection and BEV encoding, where the adaptive feature selection fadaptive(F) = σ (Wfavg(F)) · F. The independent encoders for different tasks employ the same architecture but maintain independent weights.

Model-Merging-Based Pre-training

A model-merging-based pre-training strategy is introduced to further enhance multitask accuracy. This strategy consolidates the encoder weights from multiple single-task models into a single encoder and uses it as a pre-trained initialization. Experiments show that when using a 3D object detection model to initialize the encoder, the resulting multi-task model achieves higher accuracy in 3D object detection, while the accuracy of other tasks decreases.

End-to-End Autonomous Driving

The framework is extended into an end-to-end autonomous driving model that employs an attention-based worldprediction module to perform prediction and ego vehicle trajectory planning simultaneously. The prediction and planning decoder employs N Transformer layers for iterative prediction, where the initial query is defined as Q0 = ⟨F0, F1,2,…,K⟩. The total loss is a weighted sum L =αdepthLdepth + αclsLcls+αbboxLbbox + αsegLseg + αoccLocc. The end-to-end model introduces three additional losses: the L1 loss between the ego vehicle’s future trajectory and the ground truth, denoted as Lplan, and the collision constraint Lcol.

Conclusion

HENet++ achieves state-of-the-art end-toend multi-task perception performance on the nuScenes dataset. Based on the HENet++ perception framework, it is the first work that leverages Radar and Camera for end-toend autonomous driving. On the nuScenes dataset, the HENet++ model achieves a lower collision rate compared to existing methods.

How it works

The framework is structured in three stages: I) Hybrid Image Encoding Network, II) Temporal Feature Integration, and III) Independent BEV Feature Encoding. The Hybrid Image Encoding Network uses image encoders of varying complexity to encode long-sequence frames and short-term images, respectively.

Hybrid Encoding for Sparse Instance Feature

The framework innovatively introduces the simultaneous extraction of sparse foreground features and dense background voxel features by extending hybrid encoding to be compatible with both sparse and dense feature extraction. This process involves two backbones and FPNs to extract 2D feature maps from short-term high-resolution images and long-term low-resolution images, further deriving sparse instance features and dense voxel features from them.

Pretrain based on Model Merging

The model merging method is outlined in Algorithm 2, which combines multiple single-task models into a single encoder and uses it as a pre-trained initialization. For the weight matrices in convolutional networks and linear layers, Regression Mean is applied.

Loss for End-toEnd Autonomous Driving of HENet++

The total loss is L =αdepthLdepth + αclsLcls + αbboxLbbox + αsegLseg + αoccLocc. The end-to-end model introduces three additional losses: the L1 loss between the ego vehicle’s future trajectory and the ground truth, denoted as Lplan, and the collision constraint Lcol.

Final Results Comparison

HENet++ outperforms BEVFormer [20] by 11.0 NDS and 15.5 mAP on the 3D object detection task, 8.9 mIoU on the BEV semantic segmentation task, and 15.9 mIoU on the Occupancy task. HENet++ achieves state-of-the-art end-toend multi-task perception performance on the nuScenes dataset. The HENet++ end-toend autonomous driving model retained the top 200 detection results and downsampled the voxels to 50 × 50 × 4. The HENet++ end-toend autonomous driving model achieves a lower collision rate than existing methods.

Ablation Study Summary

The ablation study demonstrates that by combining with BEVDepth4D [13] and BEVStereo [15] through hybrid image encoding, HENet can significantly improve the 3D object detection performance. The proposed backward and forward processes with AFFM achieve the best results in temporal feature integration. The independent adaptive feature selection and BEV encoder for each task can further improve the multi-task performance of 1.7 NDS, 1.1 mAP, and 3.4 mIoU. The model merging strategy improves multi-task performance compared to using single-task detection or occupancy weights. The total loss is L =αdepthLdepth + αclsLcls + αbboxLbbox + αsegLseg + αoccLocc.

Conclusion

In this paper, we first present HENet, an end-toend framework for multi-task 3D perception. We propose a Hybrid Image Encoding Network for BEV and a Temporal Feature Integration Module to handle high-resolution, long-term temporal image inputs efficiently. Based on the HENet++ perception framework, the model leverages Radar and Camera for end-toend autonomous driving. The HENet++ model achieves state-of-the-art end-toend multi-task perception performance on the nuScenes dataset. Based on the HENet++ perception framework, it is the first work that leverages Radar and Camera for end-toend autonomous driving. On the nuScenes dataset, the HENet++ model achieves a lower collision rate compared to existing methods. The framework is compatible with both sparse and dense features, enabling relatively straightforward integration with existing multimodal feature fusion methods.

References

[1] Weng, X., Ivanovic, B., Wang, Y., Wang, Y., Pavone, M.: Para-drive: Parallelized architecture for real-time autonomous driving. In: CVPR, pp. 15449–15458 (2024)

[2] Zheng, W., Song, R., Guo, X., Chen, L.: Genad: Generative end-to-end autonomous driving.

1. The HENet++ framework can simultaneously extract sparse foreground features and dense background voxel features, which enables end-to-end prediction for 3D object detection, BEV semantic segmentation, and occupancy semantic segmentation. This allows the system to leverage more comprehensive information for prediction and planning by providing task-specific representations.

2. The model can perform end-to-end autonomous driving by utilizing an attention-based world-prediction module where "instance features (including the ego vehicle’s features) as queries, and instance features combined with panoramic dense features as key-value pairs, enabling simultaneous iterative prediction of future states and ego planning."

3. The system can incorporate multimodal inputs by leveraging the framework's compatibility with both sparse and dense features to integrate Radar Encoder from RCBEVDet++ [71] along with the interpolation method described in the paper, allowing for end-to-end autonomous driving that uses both Radar and Camera.

4. The model can improve multi-task accuracy through a model-merging based pre-training strategy by consolidating encoder weights from single-task models, which is shown to achieve better performance than loading parameters from 2D tasks alone.

5. The system can achieve lower collision rates on the nuScenes benchmark by employing an end-to-end loss function that includes the L1 loss between the ego vehicle’s future trajectory and the ground truth, denoted as Lplan, alongside prediction and collision constraints (Lcol).