Abstract: In constrained scenarios of underground coal mines, object detection accuracy is low due to complex object scale variation, partial occlusion of objects, and difficulty in extracting effective features. To address these problems, an improved YOLOv8n-based object detection algorithm for constrained scenarios in underground coal mines was proposed. In the backbone feature extraction network, Receptive Field Attention Convolution (RFAConv) was adopted to better process the spatial location information of objects in constrained environments and was used to dynamically adjust weights according to the importance of features, thereby focusing more on the key features of objects. In the neck, the Efficient Multiscale Attention (EMA) module was introduced to fuse feature information at different scales, which improved the detection accuracy of objects with scale variation. The new Deformable Convolutional Networks v3 (DCNv3) and Dynamic Head were combined, integrating scale-aware attention, spatial-aware attention, and task-aware attention, which helped the model focus on spatial scale information and adapt to different detection tasks, thus enhancing the detection ability for multi-scale and partially occluded objects. A Unified-IoU (U-IoU) loss function that considers the weight distribution of prediction boxes was introduced. By dynamically adjusting the attention on prediction boxes of different qualities, the model focused more on high-quality prediction boxes, improving the convergence speed and accuracy. Experimental results showed that the improved YOLOv8n achieved a 5.6% improvement in mAP@0.5 compared with YOLOv8n in underground coal mine conveyor belt foreign object detection for the CUMT-BelT dataset; in different fully mechanized mining face operation scenarios, the overall mAP@0.5 increased by 4.8% compared with YOLOv8n for the DsLMF dataset, effectively reducing false detections and duplicate detections.