一种基于摄像头和毫米波雷达的多模态信息融合算法
Multi-Modal Information Fusion Algorithm Based on Camera and Millimeter Wave Radar
摘要: 环境感知是高级驾驶辅助系统(Advanced Driver Assistance Systems, ADAS)的关键环节，而摄像头和毫米波雷达是环境感知的核心传感器。利用多源传感器不同模态的互补信息，可以显著提升车辆的自主感知能力，帮助车辆更好地应对复杂场景下的目标检测任务。由于毫米波雷达点云的稀疏性，现有的摄像头和毫米波雷达融合算法，存在对毫米波雷达点云信息利用不充分、缺乏鲁棒性等特点。针对这些问题，本文提出了一种基于点云膨胀的数据级融合算法。算法首先使用提出的最近邻帧同步算法以及空间坐标映射进行多源数据时空对齐。之后，使用提出的基于点云膨胀的增强中心融合网络(Enhanced Center Fusion Net, ECFN)将映射至像素坐标系的毫米波雷达数据进行特征增强，并引入1 × 1的卷积核对输入数据进行降维，实现跨通道的信息交互。此外，ECFN还在损失函数中引入新的速度和深度因子来增强神经网络对雷达点云信息的利用。实验结果表明，增强的中心融合网络ECFN在推理时间稍有增加的情况下，平均精度优于基于单源传感器的算法以及现有的多源融合网络。
Abstract: Environmental awareness is the key link of Advanced Driver Assistance Systems (ADAS), and camera and millimeter wave radar are the core sensors of environmental awareness. Using the complementary information of different modes of multi-source sensors can significantly improve the autonomous sensing ability of vehicles, and help vehicles better cope with the target detection task in complex scenes. Due to the sparsity of the millimeter wave radar point cloud, the existing camera and millimeter wave radar fusion algorithms do not make full use of the millimeter wave radar point cloud information and lack of robustness. Aiming at these problems, this paper proposes a data-level fusion algorithm based on point cloud expansion. The algorithm first uses the proposed nearest neighbor frame synchronization algorithm and spatial coordinate mapping to align mul-ti-source data in time and space. Then, the proposed Enhanced Center Fusion Net (ECFN) based on point cloud expansion is used to enhance the features of millimeter wave radar data mapped to the pixel coordinate system, and the 1 × 1 convolution kernel is introduced to reduce the dimension of the input data to realize cross-channel information interaction. In addition, ECFN also introduces new velocity and depth factors into the loss function to enhance the use of radar point cloud information by neural networks. The experimental results show that the average accuracy of the enhanced central fusion network ECFN is better than that of the algorithm based on a single source sensor and the existing multi-source fusion network when the inference time is slightly increased.
文章引用：刘振东, 宋春林. 一种基于摄像头和毫米波雷达的多模态信息融合算法[J]. 计算机科学与应用, 2022, 12(11): 2544-2560. https://doi.org/10.12677/CSA.2022.1211260

参考文献

[1] 《中国公路学报》编辑部. 中国汽车工程学术研究综述∙2017 [J]. 中国公路学报, 2017, 30(6): 1-197. [Google Scholar] [CrossRef] 
[2] Wei, Z., Zhang, F., Chang, S., et al. (2022) MmWave Radar and Vision Fusion for Object Detection in Autonomous Driving: A Review. Sensors, 22, Article 2542. [Google Scholar] [CrossRef] [PubMed]
[3] Langer, D. and Jochem, T. (1996) Fusing Radar and Vision for Detecting, Classifying and Avoiding Roadway Obstacles. Proceedings of Conference on Intelligent Vehicles IEEE, Tokyo, 19-20 September 1996, 333-338. [Google Scholar] [CrossRef] 
[4] Chavez-Garcia, R.O., Burlet, J., Vu, T.-D. and Aycard, O. (2012) Frontal Object Perception Using Radar and Mono-Vision. 2012 IEEE Intelligent Vehicles Symposium, Madrid, 3-7 June 2012, 159-164. [Google Scholar] [CrossRef] 
[5] 张炳力, 詹叶辉, 潘大巍, 等. 基于毫米波雷达和机器视觉融合的车辆检测[J]. 汽车工程, 2021, 43(4): 478-484.
[6] Coué, C., Fraichard, T., Bessiere, P. and Mazer, E. (2002) Multi-Sensor Data Fusion Using Bayesian Programming: An Automotive Application. IEEE/RSJ International Confer-ence on Intelligent Robots and Systems, Lausanne, 30 September-4 October 2002, 141-146. [Google Scholar] [CrossRef] 
[7] Kawasaki, N. and Kiencke, U. (2004) Standard Platform for Sensor Fusion on Advanced Driver Assistance System Using Bayesian Network. IEEE Intelligent Vehicles Symposium, Parma, 14-17 June 2004, 250-255. [Google Scholar] [CrossRef] 
[8] Ćesić, J., Marković, I., Cvišić, I. and Petrović, I. (2016) Radar and Stereo Vision Fusion for Multitarget Tracking on the Special Euclidean Group. Robotics and Autonomous Systems, 83, 338-348. [Google Scholar] [CrossRef] 
[9] Zhong, Z., Liu, S., Mathew, M. and Dubey, A. (2018) Camera Radar Fusion for Increased Reliability in ADAS Applica-tions. Electronic Imaging, 30, art00011. [Google Scholar] [CrossRef] 
[10] Kim, D.Y. and Jeon, M. (2014) Data Fusion of Radar and Image Measurements for Multi-Object Tracking via Kalman Filtering. Information Sciences, 278, 641-652. [Google Scholar] [CrossRef] 
[11] Han, Z.B., Zhang, C.Q., Fu, H.Z. and Zhou, T.Y. (2021) Trusted Multi-View Classification. ArXiv, 2102.02051. [Google Scholar] [CrossRef] 
[12] Obrvan, M., Ćesić, J. and Petrović, I. (2016) Appearance Based Vehicle Detection by Radar-Stereo Vision Integration. In: Reis, L.P., et al., Eds., Robot 2015: Second Iberian Robotics Conference, Springer, Cham, 437-449. [Google Scholar] [CrossRef] 
[13] Wu, S., Decker, S., Chang, P., Camus, T. and Eledath, J. (2009) Collision Sensing by Stereo Vision and Radar Sensor Fusion. IEEE Transactions on Intelligent Transportation Systems, 10, 606-614. [Google Scholar] [CrossRef] 
[14] Chadwick, S., Maddern, W. and Newman, P. (2019) Distant Ve-hicle Detection Using Radar and Vision. 2019 International Conference on Robotics and Automation (ICRA) IEEE, Montreal, 20-24 May 2019, 8311-8317. [Google Scholar] [CrossRef] 
[15] Chang, S., Zhang, Y., Zhang, F., et al. (2020) Spatial Attention Fusion for Obstacle Detection Using Mmwave Radar and Vision Sensor. Sensors, 20, Article 956. [Google Scholar] [CrossRef] [PubMed]
[16] Yadav, R., Vierling, A. and Berns, K. (2020) Radar + RGB Attentive Fusion for Robust Object Detection in Autono-mous Vehicles. IEEE International Conference on Image Processing (ICIP), Abu Dhabi, 25-28 October 2020, 1986-1990. [Google Scholar] [CrossRef] 
[17] 黄昌霸. 车载毫米波雷达目标检测技术研究[D]: [硕士学位论文]. 成都: 电子科技大学, 2020.[CrossRef] 
[18] Wang, Z., Wu, Y. and Niu, Q. (2019) Multi-Sensor Fu-sion in Automated Driving: A Survey. IEEE Access, 8, 2847-2868. [Google Scholar] [CrossRef] 
[19] Guo, X.-P., Du, J.-S., Gao, J. and Wang, W. (2018) Pedes-trian Detection Based on Fusion of Millimeter Wave Radar and Vision. Proceedings of the 2018 International Confer-ence on Artificial Intelligence and Pattern Recognition, Beijing, 18-20 August 2018, 38-42. [Google Scholar] [CrossRef] 
[20] 罗逍, 姚远, 张金换. 一种毫米波雷达和摄像头联合标定方法[J]. 清华大学学报(自然科学版), 2014, 54(3): 289-293. [Google Scholar] [CrossRef] 
[21] Nobis, F., Geisslinger, M., Weber, M., Betz, J. and Lienkamp, M. (2019) A Deep Learning-Based Radar and Camera Sensor Fusion Architecture for Object Detection. 2019 Sensor Data Fusion: Trends, Solutions, Applications (SDF), Bonn, 15-17 October 2019, 1-7. [Google Scholar] [CrossRef] 
[22] Zhou, X., Wang, D. and Krähenbühl, P. (2019) Objects as Points. ArXiv, 1904.07850. [Google Scholar] [CrossRef] 
[23] Nabati, R. and Qi, H. (2021) Centerfusion: Center-Based Radar and Camera Fusion for 3d Object Detection. Proceedings of the IEEE/CVF Winter Conference on Applications of Com-puter Vision, Waikoloa, 3-8 January 2021, 1527-1536. [Google Scholar] [CrossRef] 
[24] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. and Wojna, Z. (2016) Rethinking the Inception Architecture for Computer Vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, 27-30 June 2016, 2818-2826. [Google Scholar] [CrossRef] 
[25] Lin, M., Chen, Q. and Yan, S. (2013) Network in Network. ArXiv, 1312.4400. [Google Scholar] [CrossRef] 
[26] Caesar, H., Bankiti, V., Lang, A.H., et al. (2020) nuScenes: A Multimodal Dataset for Autonomous Driving. Proceedings of the IEEE/CVF Conference on Computer Vision and Pat-tern Recognition, Seattle, 13-19 June 2020, 11621-11631. [Google Scholar] [CrossRef]