CMRNet: Camera to LiDAR-Map Registration

Abstract. In this paper we present CMRNet, a realtime approach based on a Convolutional Neural Network (CNN) to localize an RGB image of a scene in a map built from LiDAR data. Our network is not trained in the working area, i. e., CMRNet does not learn the map. Instead it learns to match an image to the map. We validate our approach on the KITTI dataset, processing each frame independently without any tracking procedure. CMRNet achieves 0.27m and 1.07° median localization accuracy on the sequence 00 of the odometry dataset, starting from a rough pose estimate displaced up to 3.5m and 17°. To the best of our knowledge this is the first CNN-based approach that learns to match images from a monocular camera to a given, preexisting 3D LiDAR-map.

Download paper here

Recommended citation: @INPROCEEDINGS{8917470, author={D. {Cattaneo} and M. {Vaghi} and A. L. {Ballardini} and S. {Fontana} and D. G. {Sorrenti} and W. {Burgard}}, booktitle={2019 IEEE Intelligent Transportation Systems Conference (ITSC)}, title={CMRNet: Camera to LiDAR-Map Registration}, year={2019}, volume={}, number={}, pages={1283-1289}, keywords={cameras;convolutional neural nets;image colour analysis;image matching;image registration;learning (artificial intelligence);optical radar;pose estimation;CMRNet;LiDAR-map registration;Convolutional Neural Network;RGB image;LiDAR data;working area;KITTI dataset;CNN-based approach;3D LiDAR-map;monocular camera;median localization accuracy;distance 3.5 m;Cameras;Three-dimensional displays;Laser radar;Task analysis;Machine learning;Quaternions;Urban areas}, doi={10.1109/ITSC.2019.8917470}, ISSN={null}, month={Oct},}