Abstract: Small object detection in open-pit mines faces challenges such as wide viewing angles and long detection distances, which result in small target imaging. Existing object detection models suffer from feature attenuation caused by progressive image downsampling operations. To address this issue, an improved YOLOv11 model was proposed and applied to small object detection under complex backgrounds in open-pit mines. The improved YOLOv11 model introduced a Robust Feature Downsampling (RFD) module to replace the stride convolution downsampling module, effectively preserving the feature information of small objects. A Small Target Feature Enhancement Neck (STFEN) network was designed to replace the original feature pyramid structure in the neck, incorporating a cross-stage partial fusion module to integrate feature maps from different levels. The original CIoU loss function was replaced with the Powerful-IoU (PIoU) loss function to solve the anchor box expansion issue during training, enabling the model to rapidly and accurately focus on small targets. Experimental results on a small object dataset from open-pit mining areas showed that: ① the RFD module reduced model parameters while increasing mAP by 1.5%. Although the STFEN network increased the number of parameters, it improved mAP by 2.2%. The PIoU loss function improved mAP by 1.7% without changing the number of parameters or FLOPs. The combination of all three led to a total mAP improvement of 3.9%. ② The improved YOLOv11 model achieved higher accuracy while maintaining a high inference speed, with mAP improvements of 2.6%, 1.5%, 0.9%, and 2.2% over YOLOv5m, YOLOv8m, YOLOv11m, and RtDetr-L, respectively, and with fewer parameters, making it more suitable for edge deployment.