Abstract: The existing deep learning-based coal and gangue recognition methods are prone to false detection and missed detection when applied to underground complex environments. The recognition precision of small target coal and gangue is low. In order to solve this problem, an improved YOLOv5 model is proposed, and coal and gangue recognition is realized based on that model. Data enhancement is carried out on the collected coal and gangue data to enrich the data set and improve the data utilization rate. The atrous convolution and residual block are introduced into the spatial pyramid pooling (SPP) module to obtain the residual ASPP module. On the premise of not losing image information, the convolution output receptive field can be increased to enhance the extraction of deep features from the model. The AdaBelief optimization algorithm is used to replace the original Adam optimization algorithm of YOLOv5 to improve the convergence speed and recognition precision of the model. The experimental results show that the AdaBelief optimization algorithm and residual ASPP module can effectively improve the precision, recall rate and mean average precision (mAP) of the YOLOv5 model. The mAP of the improved YOLOv5 model reaches 94.43%, which is 2.27% higher than that of original YOLOv5 model. The frame rate is reduced by 0.03 frames/s. The performance of the improved YOLOv5 model is superior to SSD, Faster R-CNN, YOLOv3, YOLOv4 and other mainstream target detection models. In extremely dark environments, the improved YOLOv5 model can also accurately delineate the target boundary, and the recognition effect is better than other improved YOLOv5 models.