Abstract: To address the issues of inaccurate judgment, numerous false positives and false negatives, and limited real-time performance in personnel intrusion detection under complex poses in underground mine conveyor belt hazardous areas, an improved YOLOv5-seg personnel intrusion detection method is proposed. Based on YOLOv5-seg, the network structure is improved by reducing the number of model parameters and computational cost through a lightweight neck design, and the Proto branch is optimized to enhance the human mask reconstruction capability and improve the segmentation quality of complex poses and edge regions. Furthermore, a candidate buffer triggering mechanism is introduced, performing fine segmentation only on targets approaching the hazardous area to reduce unnecessary computational overhead and improve the overall inference efficiency of the system. On this basis, a dynamic safety boundary based on geometric constraints is constructed, and the overlap relationship between the human segmentation mask and the ROI of the hazardous area is used to realize intrusion judgment, thereby improving the accuracy and reliability of hazardous area boundary crossing identification. Based on a self-built dataset of 3000 images of hazardous areas in underground conveyor belts, verification experiments, ablation experiments, and comparative experiments were conducted on the judgment method. Statistical results on the test samples show that the accuracy, precision, recall, and F1 score of the bottom edge midpoint determination method are 88.4%, 97.4%, 47.0%, and 63.4%, respectively; the box bottom edge determination method is 90.1%, 96.7%, 55.5%, and 70.5%; and the segmentation mask overlap determination method reaches 94.4%, 85.5%, 88.7%, and 87.1%, respectively. Although the false positive rate of the segmentation mask overlap determination method is 4.07%, slightly higher than the other two geometric determination methods, its false negative rate is only 11.3%, significantly lower than the 52.9% of the bottom edge midpoint determination method and the 44.5% of the box bottom edge determination method. Ablation experiments show that lightweight Neck, Proto branch optimization, and candidate buffer mechanisms can all positively improve model performance. Through the synergistic effect of these three factors, the improved model's mAP@0.5 increased from 91.9% to 93.5% compared to the baseline model, the number of parameters decreased from 7.408M to 5.723M, and the computational cost decreased from 25.91G to 14.22G, while maintaining an inference speed of 121.87 FPS, achieving a better balance between accuracy, real-time performance, and lightweight design. Comparative experimental results show that the proposed method achieves an mAP@0.5 of 93.5% and an inference speed of 122 FPS, maintaining excellent overall performance in both mAP@0.5 and inference speed. Furthermore, the mask overlap-based determination method demonstrates higher reliability in complex pose and edge contact scenarios. The results indicate that the segmentation mask overlap determination method can more accurately describe the true spatial relationship between human areas and hazardous ROIs, exhibiting superior comprehensive detection performance in complex underground scenarios. This method can provide a reference for intelligent early warning algorithm systems for hazardous areas of conveyor belts and for determining the spatial location of personnel.