Penalty-Based Imitation Learning With Cross Semantics Generation Sensor Fusion for Autonomous Driving

Hongkuan Zhou*, Aifen Sui* †, Letian Shi*, Yinxian Li
Huawei Munich Research Center, Technical University of Munich
IEEE International Conference on Intelligent Transportation Systems ITSC 2023
*Indicates Equal Contribution †Corresponding Author
Paper Code arXiv

Abstract

In recent times, there has been a growing focus on end-to-end autonomous driving technologies. This technology involves the replacement of the entire driving pipeline with a single neural network, which has a simpler structure and faster inference time. However, while this approach reduces the number of components in the driving pipeline, it also presents challenges related to interpretability and safety. For instance, the trained policy may not always comply with traffic rules, and it is difficult to determine the reason for such misbehavior due to the lack of intermediate outputs. Additionally, the successful implementation of autonomous driving technology heavily depends on the reliable and expedient processing of sensory data to accurately perceive the surrounding environment. In this paper, we provide penalty-based imitation learning approach combined with cross semantics generation sensor fusion technologies (PCSG) to efficiently integrate multiple modalities of information and enable the autonomous agent to effectively adhere to traffic regulations. Our model undergoes evaluation within the Town 05 Long benchmark, where we observe a remarkable increase in the driving score by more than 12% when compared to the state-of-the-art (SOTA) model, InterFuser. Notably, our model achieves this performance enhancement while achieving a 7- fold increase in inference speed and reducing the model size by approximately 30%.

MY ALT TEXT

The top-down LiDAR pseudo image and front camera image go through two residual networks to extract 512 dimension feature vectors. We use four different MLPs to extract the shared features and the unique features. The unique features of RGB input are used to generate stop signs and traffic light indicators. The shared features of LiDAR are used to reconstruct the segmentation of RGB input while the shared features of RGB are used to reconstruct the segmentation of top-down LiDAR input. An alignment loss is used to align the shared features from LiDAR and camera inputs into the same space. These shared features and unique features are concatenated along with the measurements (velocity, throttle, steer, brake from the last frame) and then go through one MLP to reduce the size. Finally, they will be fed into one GRU decoder to predict short-term waypoints.

MY ALT TEXT

To ensure compliance with red light and stop penalty rules, as well as promoting deceleration during turning maneuvers, our approach incorporates three distinct penalty types. The first column of the figures exemplifies the red light penalty, wherein waypoints situated beyond the stop line receive a penalty when the traffic light is red. In the second column, we demonstrate the stop sign penalty, wherein predicted waypoints within the vicinity of a stop sign are penalized if the agent fails to decelerate adequately. The speed penalty is enforced during turning actions as shown in last two figures. Specifically, if the predicted waypoints indicate an excessive speed, a speed penalty is imposed.

Experiments

Model Driving Score Route Complication Infraction Score
LateFusion 32.34 60.41 0.61
Geometric Fusion 31.30 57.17 0.54
InterFuser 50.64 89.13 0.57
Transfuser+ 36.19 70.13 0.51
Transfuser 34.50 61.16 0.56
P_CSG 56.38 94.00 0.61

BibTeX


        @misc{zhou2023penaltybased,
              title={Penalty-Based Imitation Learning With Cross Semantics
              Generation Sensor Fusion for Autonomous Driving},
              author={Hongkuan Zhou and Aifen Sui and Letian Shi and Yinxian Li},
              year={2023},
              eprint={2303.11888},
              archivePrefix={arXiv},
              primaryClass={cs.RO}
        }

Address

Riesstraße 25, 80992 München

Contact

aifen.sui@huawei.com