Video object detection of the fully mechanized working face based on deep neural network
-
摘要: 综采工作面环境较复杂，地形狭长，多目标多设备经常出现在同一场景当中，使得目标检测难度加大。目前应用于煤矿井下的目标检测方法存在特征提取难度较大、泛化能力较差、检测目标类别较为单一等问题，且主要应用于巷道、井底车场等较为空旷场景，较少应用于综采工作面场景。针对上述问题，提出了一种基于深度神经网络的综采工作面视频目标检测方法。首先，针对综采工作面环境复杂多变、光照不均、煤尘大等不利条件，针对性挑选包含各角度、各环境条件下的综采工作面关键设备和人员的监控视频，并进行剪辑、删选，制作尽可能涵盖工作面现场各类场景的目标检测数据集。然后，通过对 YOLOv4模型进行轻量化改进，构建了LiYOLO目标检测模型。该模型利用CSPDarknet、SPP、PANet等加强特征提取模块对视频特征进行充分提取，使用6分类YoloHead进行目标检测，对综采工作面环境动态变化、煤尘干扰等具有较好的鲁棒性。最后，将LiYOLO目标检测模型部署到综采工作面，应用Gstreamer对视频流进行管理，同时使用TensorRT对模型进行推理加速，实现了多路视频流的实时检测。与YOLOv3、YOLOv4模型相比，LiYOLO目标检测模型具有良好的检测能力，能够满足综采工作面视频目标检测的实时性和精度要求，在综采工作面数据集上的平均准确率均值为96.48%，召回率为95%，同时视频检测帧率达67帧/s。工程应用效果表明，LiYOLO目标检测模型可同时检测、展示6路视频，且对于不同场景下的检测目标都有较好的检测效果。Abstract: The environment of the fully mechanized working face is complex. The terrain is long and narrow. The multi-object and multi-equipment often appear in the same scene, which makes object detection more difficult. At present, the object detection method applied to the underground coal mine has the problems of high difficulty in characteristic extraction, poor generalization capability, and relatively single detection object category. The existing method is mainly applied to open scenes such as a roadway, a shaft bottom station, and is rarely applied to scenes of a fully mechanized working face. In order to solve the above problems, a video object detection method based on deep neural network is proposed. Firstly, in view of the unfavorable conditions such as complex and changeable environments, uneven illumination, and much coal dust in the fully mechanized working face, the monitoring videos are selected which containing key equipment and personnel of the fully mechanized working face at various angles and under various environmental conditions. By editing, deleting and selecting, an object detection data set covering various scenes of the working face site as much as possible is produced. Secondly, the LiYOLO object detection model is constructed by lightweight improvement of YOLOv4 model. The model fully extracts video characteristics by using CSPDarknet, SPP, PANet and other enhanced characteristic extraction modules. This model uses 6-classification YoloHead for object detection, which has good robustness to the dynamic change of environment and coal dust interference in fully mechanized working face. Finally, the LiYOLO object detection model is deployed to the fully mechanized working face. While the video stream is managed by Gstreamer, TensorRT is used to accelerate the reasoning of the model, and realize the real-time detection of multi-channel video streams. Compared with the YOLOv3 and YOLOv4 models, the LiYOLO object detection model has good detection capability, and can meet the real-time and precision requirements of video object detection in the fully mechanized working face. The mean average precision on the data set of fully mechanized working face is 96.48%, the recall rate is 95%, and the frame rate of video detection can reach 67 frames/s. The engineering application results show that the LiYOLO object detection model can detect and display 6-channel videos at the same time. The model has relatively good detection effect for detection of objects in different scenes.
-
图 1 综采工作面视频目标检测流程
Figure 1. Flow of video object detection in fully mechanized working face
图 2 不同条件下的综采工作面图像
Figure 2. Images of fully mechanized working face under different conditions
图 8 3种模型对不同场景下设备及行人的检测效果
Figure 8. Detection effect of three models for devices and pedestrians in different scenes
表 1 不同条件下的图像采集数量
Table 1. Number of image samples under different conditions
位置 环境 图像采集数量/张 无尘 轻微 严重 端头 顺光 4 524 2 263 2 263 逆光 1 132 565 565 中部 顺光 22 622 11 313 11 313 逆光 5 657 2 828 2 828 
下载: 导出CSV
表 2 标签分类
Table 2. Classification of labels
序号 标签名称 序号 标签名称 1 Groove(线槽) 4 Roller(滚筒) 2 Conveyer(刮板输送机) 5 Person(人) 3 Shearer(采煤机) 6 face_guard(护帮板)
下载: 导出CSV
表 3 主要实验结果对比
Table 3. Comparison of main experimental results
% 模型 mAP Recall YOLOv4 81.69 90 LiYOLO 96.48 95
下载: 导出CSV
表 4 检测时间
Table 4. Detection time
模型 检测时间/ms 传输帧率/（帧·s−1） YOLOv3 29.9 27.9 YOLOv4 16.2 59.1 LiYOLO 16.1 66.8
下载: 导出CSV
表 5 未加速与加速后模型FPS对比
Table 5. Comparison of FPS between the unaccelerated model and the accelerated model
帧/s 未加速FPS 加速后FPS 1路 1路 4路 6路 55.2 85.4 20.8×4 13.9×6
下载: 导出CSV
-
[1] 王国法,刘峰,庞义辉,等. 煤矿智能化−煤炭工业高质量发展的核心技术支撑[J]. 煤炭学报,2019,44(2):349-357. doi: 10.13225/j.cnki.jccs.2018.2041WANG Guofa,LIU Feng,PANG Yihui,et al. Coal mine intellectualization:the core technology of high quality development[J]. Journal of China Coal Society,2019,44(2):349-357. doi: 10.13225/j.cnki.jccs.2018.2041 [2] 高有进,杨艺,常亚军,等. 综采工作面智能化关键技术现状与展望[J]. 煤炭科学技术,2021,49(8):1-22. doi: 10.13199/j.cnki.cst.2021.08.001GAO Youjin,YANG Yi,CHANG Yajun,et al. Status and prospect of key technologies of intelligentization of fully mechanized coal mining face[J]. Coal Science and Technology,2021,49(8):1-22. doi: 10.13199/j.cnki.cst.2021.08.001 [3] 王道元,王俊,孟志斌,等. 煤矿安全风险智能分级管控与信息预警系统[J]. 煤炭科学技术,2021,49(10):136-144. doi: 10.13199/j.cnki.cst.2021.10.019WANG Daoyuan,WANG Jun,MENG Zhibin,et al. Intelligent hierarchical management and control and information pre-warning system of coal mine safety risk[J]. Coal Science and Technology,2021,49(10):136-144. doi: 10.13199/j.cnki.cst.2021.10.019 [4] 郭金刚,李化敏,王祖洸,等. 综采工作面智能化开采路径及关键技术[J]. 煤炭科学技术,2021,49(1):128-138. doi: 10.13199/j.cnki.cst.2021.01.007GUO Jingang,LI Huamin,WANG Zuguang,et al. Path and key technologies of intelligent mining in fully-mechanized coal mining face[J]. Coal Science and Technology,2021,49(1):128-138. doi: 10.13199/j.cnki.cst.2021.01.007 [5] 王国法,任怀伟,庞义辉,等. 煤矿智能化(初级阶段)技术体系研究与工程进展[J]. 煤炭科学技术,2020,48(7):1-27. doi: 10.13199/j.cnki.cst.2020.07.001WANG Guofa,REN Huaiwei,PANG Yihui,et al. Research and engineering progress of intelligent coal mine technical system in early stages[J]. Coal Science and Technology,2020,48(7):1-27. doi: 10.13199/j.cnki.cst.2020.07.001 [6] 任怀伟,孟祥军,李政,等. 8 m大采高综采工作面智能控制系统关键技术研究[J]. 煤炭科学技术,2017,45(11):37-44.REN Huaiwei,MENG Xiangjun,LI Zheng,et al. Study on key technology of intelligent control system applied in 8 m large mining height fully-mechanized face[J]. Coal Science and Technology,2017,45(11):37-44. [7] DALAL N, TRIGGS B. Histograms of oriented gradients for human detection [EB/OL]. （2017-02-23）[2022-02-20]. https://blog.csdn.net/yurnm/article/details/56673837. [8] LOWE D G. Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision,2004,60(2):91-110. doi: 10.1023/B:VISI.0000029664.99615.94 [9] FELZENSZWALB P, MCALLESTER D, RAMANAN D. A discriminatively trained, multiscale, deformable part model[EB/OL]. [2022-01-20]. https://ieeexplore.ieee.org/document/4587597/footnotes#footnotes. [10] 孙继平,贾倪. 矿井视频图像中人员目标匹配与跟踪方法[J]. 中国矿业大学学报,2015,44(3):540-548. doi: 10.13247/j.cnki.jcumt.000264SUN Jiping,JIA Ni. Human target matching and tracking method in coal mine video[J]. Journal of China University of Mining & Technology,2015,44(3):540-548. doi: 10.13247/j.cnki.jcumt.000264 [11] 徐美华,龚露鸣,郭爱英,等. 基于自适应CtF DPM特征提取的快速行人检测模型[J]. 复旦大学学报(自然科学版),2018,57(4):453-461.XU Meihua,GONG Luming,GUO Aiying,et al. A fast pedestrian detection model based on adaptive CtF DPM feature extraction[J]. Journal of Fudan University(Natural Science),2018,57(4):453-461. [12] 张银萍. 煤矿地面轨道运输环境感知系统研究[D]. 徐州: 中国矿业大学, 2020.ZHANG Yinping. Study on environmental perception system of coal mine ground rail transportation[D]. Xuzhou: China University of Mining and Technology, 2020. [13] 卢万杰,付华,赵洪瑞. 基于深度学习算法的矿用巡检机器人设备识别[J]. 工程设计学报,2019,26(5):527-533. doi: 10.3785/j.issn.1006-754X.2019.05.005LU Wanjie,FU Hua,ZHAO Hongrui,et al. Equipment recognition of mining patrol robot based on deep learning algorithm[J]. Chinese Journal of Engineering Design,2019,26(5):527-533. doi: 10.3785/j.issn.1006-754X.2019.05.005 [14] 林俊,党伟超,潘理虎,等. 基于计算机视觉的井下输送带跑偏检测方法[J]. 煤矿机械,2019,40(10):169-171. doi: 10.13436/j.mkjx.201910057LIN Jun,DANG Weichao,PAN Lihu,et al. Deviation monitoring method of underground conveyor belt based on computer vision[J]. Coal Mine Machinery,2019,40(10):169-171. doi: 10.13436/j.mkjx.201910057 [15] 董昕宇,师杰,张国英. 基于参数轻量化的井下人体实时检测算法[J]. 工矿自动化,2021,47(6):71-78. doi: 10.13272/j.issn.1671-251x.2021010035DONG Xinyu,SHI Jie,ZHANG Guoying. Real-time detection algorithm of underground human body based on lightweight parameters[J]. Industry and Mine Automation,2021,47(6):71-78. doi: 10.13272/j.issn.1671-251x.2021010035 [16] 南柄飞, 郭志杰, 王凯, 等. 基于视觉显著性的煤矿井下关键目标对象实时感知研究[J/OL]. 煤炭科学技术: 1-11[2022-07-15]. http://kns.cnki.net/kcms/detail/11.2402.TD.20210512.1304.004.html.NAN Bingfei, GUO Zhijie, WANG Kai, et al. Real-time perception method of target ROI in coal mine underground based on visual saliency[J/OL]. Coal Science and Technology: 1-11[2022-07-15]. http://kns.cnki.net/kcms/detail/11.2402.TD.20210512.1304.004.html. [17] 韩江洪,沈露露,卫星,等. 基于轻量级CNN的井下视觉识别策略[J]. 合肥工业大学学报(自然科学版),2020,43(11):1469-1475,1562.HAN Jianghong,SHEN Lulu,WEI Xing,et al. Downhole visual recognition strategy based on lightweight CNN[J]. Journal of Hefei University of Technology(Natural Science),2020,43(11):1469-1475,1562. [18] BOCHKOVSKIY A, WANG C Y, LIAO H Y M. YOLOv4: optimal speed and accuracy of object detection[EB/OL]. [2022-01-20]. https://doi.org/10.48550/arXiv.2004.10934. [19] REDMON J, FARHADI A. YOLO9000: better, faster, stronger[EB/OL]. [2022-01-22]. https://wenku.baidu.com/view/d74b46407b3e0912a21614791711cc7931b778d6.html. [20] HE Kaiming,ZHANG Xiangyu,REN Shaoqing,et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2014,37(9):1904-1916. [21] LIU S, QI L, QIN H F, et al. Path aggregation network for instance segmentation[EB/OL]. [2022-01-15]. https://ieeexplore.ieee.org/document/8579011. -
点击查看大图
图(10) / 表(5)
计量
- 文章访问数: 326
- HTML全文浏览量: 50
- PDF下载量: 67
- 被引次数: 0
