Obstacle detection method for autonomous driving in open-pit mines based on fusion of point cloud and image features

At present, obstacle perception during the driving process of autonomous mining trucks in open-pit mines is mostly based on a single LiDAR point cloud or camera image features. Affected by point cloud noise and low-quality images, the detection accuracy and reliability are limited. Existing point cloud and image feature fusion detection methods fail to effectively address the heterogeneous alignment problem between sparse point clouds and dense images, and dense convolution operations easily lead to the loss of key point cloud features. To address this problem, an obstacle detection method for autonomous driving in open-pit mines based on the fusion of point cloud and image features was proposed. Voxel R-CNN and YOLOv5 were respectively adopted to extract LiDAR point cloud features and camera image features. A focal sparse convolution network was used to fuse the two types of features. Obstacles were identified based on the fused features, and their orientation and distance were determined using the target 3D detection boxes. Experimental results showed that, compared with single-modality feature-based detection methods such as Voxel R-CNN and YOLOv5, the proposed method achieved better precision, recall, bbox accuracy, and 3D accuracy, and reduced cases of missed detections or false detections caused by single-modality feature-based detection methods. Compared with object-level fusion model and sensor-level fusion model, this method achieves a better balance between detection accuracy and real-time performance, making it more suitable for obstacle detection scenarios of autonomous driving in open-pit mines.