Abstract: The existing deep learning based coal gangue recognition methods have problems in complex working conditions such as low lighting, high noise, and motion blur in coal mines, such as low precision of coal gangue recognition, easy omission of small target coal gangue, large model parameter and computational complexity, and difficulty in deploying to devices with limited computing resources. A coal gangue recognition algorithm based on the HGTC-YOLOv8n model is proposed. The method replaces the backbone network of YOLOv8n with HGNetv2 network, effectively extracts multi-scale features to improve coal gangue recognition performance and reduces model storage requirements and computational resource consumption. The method embeds a Triplet Attention mechanism module in the backbone network to capture interaction information between different dimensions. The method enhances the extraction of target features in coal gangue images, and reduces the interference of irrelevant information. The method selects the content aware reassembly of features(CARAFE) to improve the upsampling operator of YOLOv8n neck feature fusion network, utilizing contextual information to enhance perceptual field of view and improve the accuracy of small target coal gangue recognition. The experimental results show the following points.① The average precision of the HGTC-YOLOv8n model is 93.5%, the parameters number of the model is 2.645×10⁶, the number of floating-point operation is 8.0×10⁹, and the frame rate is 79.36 frames/s. ② The average precision of the YOLOv8n model has increased by 2.5% compared to the YOLOv8n model, and the number of parameters and floating-point operations have decreased by 16.22% and 10.11%, respectively. ③ The comparison results with the YOLO series models show that the HGTC-YOLOv8n model has the highest average precision, the least number of parameters and floating-point operations, fast detection speed, and the best overall detection performance. ④ The coal gangue recognition algorithm based on the HGTC-YOLOv8n model has improved the low precision of coal gangue recognition and the easy omission of small target coal gangue under complex working conditions in coal mines. The method meets the requirements of real-time detection of coal gangue images.