Abstract: Uneven illumination, diffuse dust, multiple occlusions, and complex background textures in underground coal mine environments lead to low accuracy in safety helmet wearing detection for personnel based on computer vision technology, while the limited computing power of underground explosion-proof edge computing terminals imposes high requirements on detection model size and inference efficiency. To address this problem, an improved YOLOv11 model integrating frequency-domain enhancement and efficient lightweight mechanisms was proposed. A C3k2_WTConv module based on wavelet convolution was introduced into the YOLOv11 backbone, and multiresolution analysis was used to decouple illumination noise and texture, thereby enhancing feature extraction capability. A SlimNeck feature fusion network based on the GSConv lightweight operator and VoVGSCSP topology was constructed in the neck network to reduce computational redundancy while maintaining cross-scale feature interaction. A parameter-sharing Detect_Efficient detection head was designed to improve inference efficiency. Parameter smoothing based on exponential moving average and a multi-source domain adaptation transfer learning strategy were adopted to solve the scarcity of violation samples under extreme underground working conditions, enhancing the cross-domain generalization capability and robustness of the model in unstructured environments. The improved YOLOv11 model was used for safety helmet wearing detection of underground coal mine personnel. Verification on the CUMT−Helmet and DsLMF+Helmet datasets showed that the mAP@0.5 of the model reached 97.5%, outperforming mainstream single-stage detection models YOLOv7-tiny, YOLOv8s, and YOLOv11 and improved models MH−YOLO and WAM−YOLO; the single-frame processing time was only 10.2 ms, and the model exhibited higher confidence and lower missed detection rates under extreme working conditions such as strong light interference, small-scale distant targets, and dynamic target blur.