基于改进YOLOv5s+DeepSORT的煤块行为异常识别

张旭辉; 闫建星; 张超; 万继成; 王利欣; 胡成军; 王力; 王东

doi:10.13272/j.issn.1671-251x.17915

基于改进YOLOv5s+DeepSORT的煤块行为异常识别

1.
西安科技大学机械工程学院, 陕西西安　710054
2.
中煤(天津)地下工程智能研究院, 天津　300121
3.
陕西煤业化工集团孙家岔龙华矿业有限公司, 陕西榆林　719314

基金项目: 国家自然科学基金项目（51834006）；陕西省重点研发计划项目（2018ZDCXL-GY-06-04）。

详细信息

作者简介:
张旭辉(1972—),男，陕西凤翔人，教授，博士，博士研究生导师，研究方向为煤矿机电设备智能检测与控制，E-mail：zhangxh@xust.edu.cn

通讯作者:
闫建星(1995—)，男，陕西榆林人，硕士研究生，研究方向为智能检测与控制，E-mail:yanjianxing2013@163.com

中图分类号: TD76
计量
- 文章访问数: 496
- HTML全文浏览量: 332
- PDF下载量: 92
出版历程
- 收稿日期: 2022-04-11
- 修回日期: 2022-06-09
- 网络出版日期: 2022-06-27
- 刊出日期: 2022-06-29

Coal block abnormal behavior identification based on improved YOLOv5s + DeepSORT

1.
College of Mechanical Engineering, Xi'an University of Science and Technology, Xi'an 710054, China
2.
China Coal (Tianjin) Underground Engineering Intelligence Research Institute, Tianjin 300121, China
3.
Sunjiacha Longhua Mining Co., Ltd., Shaanxi Coal Chemical Industry Group, Yulin 719314, China

摘要

摘要: 煤块检测方法主要包括传统图像检测方法和深度学习目标检测方法。传统图像检测方法检测精度不高、实时性较差、无法对堆煤进行准确判断；深度学习目标检测方法虽然可以实现实时检测，但没有对煤块的数量、滞留和堵塞状态进行识别，而且识别模型参数较多。针对上述问题，提出了一种基于改进YOLOv5s+DeepSORT的煤块行为异常识别方法。首先通过摄像头和巡检机器人采集煤矿综采工作面带式输送机上煤块视频图像，并制作数据集。然后利用MobileNetV3_YOLOv5s_AF−FPN模型进行煤块图像目标检测：通过MobileNetV3替换原始YOLOv5s主干特征提取网络，减少参数量，提高推理速度；将YOLOv5s中原有的特征金字塔网络改进为增强特征金字塔网络（AF−FPN），以提高YOLOv5s网络对多尺度煤块目标的检测性能。利用DeepSORT进行煤块多目标跟踪：将改进YOLOv5s模型检测后的煤块图像作为DeepSORT的输入进行多目标跟踪，利用DeepSORT对煤块进行状态估计、数据关联匹配和跟踪器参数更新，确定跟踪结果，并对连续跟踪的煤块进行ID编码，对当前帧的煤块数量进行计数。最后在目标跟踪器中取出连续跟踪的目标，设置距离阈值，判断其是否滞留；设置数量阈值，判断其是否堵塞，最终实现煤块滞留和堵塞行为异常识别。利用自建dkm_data2021数据集对基于改进YOLOv5s+DeepSORT的煤块行为异常识别方法的可靠性进行实验验证，结果表明：改进YOLOv5s模型相比YOLOv5s模型平均检测精度提高了1.45%，参数量减少了35.3%，推理加速了12.7%，平均漏检率降低了11.08%，平均误检率降低了11.54%；基于改进YOLOv5s+DeepSORT的煤块行为异常识别方法检测精度为80.1%，可准确识别煤块滞留、堵塞状态，验证了该方法的可靠性。
- 煤块识别 /
- 煤块目标检测 /
- 目标跟踪 /
- 异常行为识别 /
- 煤块特征提取 /
- 煤块滞留 /
- 煤块堵塞 /
- 深度学习
Abstract: Coal block detection methods mainly include traditional image detection methods and deep learning target detection methods. The traditional image detection method has low detection precision and poor real-time performance, and can not accurately determine the coal pile. Although the deep learning target detection method can achieve real-time detection, it does not identify the number, retention, and blockage of coal blocks. And there are many identification model parameters. To solve the above problems, a coal block abnormal behavior identification method based on improved YOLOv5s + DeepSORT is proposed. Firstly, video images of coal blocks on a belt conveyor in a fully mechanized coal mining face are collected by the camera and inspection robot, and data sets are made. Secondly, the MobileNetV3_YOLOv5s_AF-FPN model is used for detecting the coal image target. The original YOLOv5s backbone feature extraction network is replaced by MobileNetV3 to reduce the number of parameters and improve the reasoning speed. The original feature pyramid network in YOLOv5s is improved to AF-FPN to improve the detection performance of the YOLOv5s network for multi-scale coal targets. DeepSORT is used for multi-target tracking of coal blocks. The coal block image detected by the improved YOLOv5s is taken as the input of DeepSORT for multi-target tracking. DeepSORT is used to estimate the state of coal blocks, perform data association and matching, and update the tracker parameters to determine the tracking results. The continuously tracked coals are ID-coded, and the number of coals in the current frame is counted. Finally, the continuously tracked target is taken out from the target tracker, and a distance threshold is set. Whether the target is detained or not is determined. The quantity threshold is set to determine whether it is blocked. The identification of abnormal behavior of coal block retention and blocking state is finally realized. The reliability of the coal abnormal behavior identification method based on the improved YOLOv5s + DeepSORT is experimentally verified by using the self-built dkm_data2021 data set. The results show that compared with the YOLOv5s model, the average detection precision of the improved YOLOv5s model is improved by 1.45%, the parameter quantity is reduced by 35.3%, the reasoning is accelerated by 12.7%, the average missed detection rate is reduced by 11.08%, and the average false detection rate is reduced by 11.54%. The detection precision of coal block abnormal behavior identification method based on the improved YOLOv5s+DeepSORT is 80.1%, which can accurately identify the status of coal block retention and blockage. The result verifies the reliability of the method.
- coal block identification /
- coal block target detection /
- target tracking /
- abnormal behavior identification /
- coal block feature extraction /
- coal block retention /
- coal block blockage /
- deep learning

HTML全文

图 1 YOLOv5s网络结构

Figure 1. YOLOv5s network structure

下载: 全尺寸图片幻灯片

图 2 改进后的YOLOv5s网络结构

Figure 2. Improved YOLOv5s network structure

下载: 全尺寸图片幻灯片

图 3 Bneck网络结构

Figure 3. Bneck network structure

下载: 全尺寸图片幻灯片

图 4 MobileNetV3_YOLOv5s_AF−FPN网络结构

Figure 4. MobileNetV3_YOLOv5s_AF-FPN network structure

下载: 全尺寸图片幻灯片

图 5 AAM网络结构

Figure 5. AAM network structure

下载: 全尺寸图片幻灯片

图 6 FEM网络结构

Figure 6. FEM network structure

下载: 全尺寸图片幻灯片

图 7 煤块行为异常识别方法流程

Figure 7. Flow of coal block abnormal behavior identification method

下载: 全尺寸图片幻灯片

图 8 DeepSORT多目标跟踪算法流程

Figure 8. DeepSORT multi-target tracking algorithm flow

下载: 全尺寸图片幻灯片

图 9 YOLOv5s模型与MobileNetV3_YOLOv5s_AF−FPN模型的煤块检测效果对比

Figure 9. Comparison of coal detection effect of YOLOv5s model and MobileNetV3_YOLOv5s_AF-FPN model

下载: 全尺寸图片幻灯片

图 10 煤块正常跟踪

Figure 10. Coal block normal tracking

下载: 全尺寸图片幻灯片

图 12 煤块堵塞

Figure 12. Coal block blockage

下载: 全尺寸图片幻灯片

图 11 煤块滞留

Figure 11. Coal block retention

下载: 全尺寸图片幻灯片

表 1 MobileNetV3_Large结构

Table 1 MobileNetV3_Large structure

Input Shape	Operator	SE	AF	Stride
224²×3	Conv2d	−	HS	2
112²×16	Bneck,3×3	−	RE	1
112²×16	Bneck,3×3	−	RE	2
56²×24	Bneck,3×3	√	RE	1
56²×24	Bneck,3×3	√	RE	2
28²×40	Bneck,3×3	√	RE	1
28²×40	Bneck,3×3	−	RE	1
28²×40	Bneck,3×3	−	HS	2
14²×80	Bneck,3×3	−	HS	1
14²×80	Bneck,3×3	−	HS	1
14²×80	Bneck,3×3	−	HS	1
14²×80	Bneck,3×3	√	HS	1
14²×112	Bneck,3×3	√	HS	1
14²×112	Bneck,5×5	√	HS	1
7²×160	Bneck,5×5	√	HS	2
7²×160	Bneck,5×5	√	HS	1
7²×160	Conv2d,1×1	−	HS	1
7²×160	Pool, 7×7	−	−	1
1²×960	Conv2d,1×1	−	HS	1
1²×1280	Conv2d,1×1	−	−	1

下载: 导出CSV

表 2 特征提取网络实验对比

Table 2 Comparison of feature extraction network experiments

模型	召回率	平均精度	参数量/M	平均漏检率	平均误检率	推理时间/ms
YOLOv5s	0.785	0.821	7.09	0.334	0.026	18.9
MobileNetV3_ YOLOv5s	0.766	0.795	3.56	0.365	0.027	15.0

下载: 导出CSV

表 3 特征融合网络实验对比

Table 3 Comparison of feature fusion network experiments

模型	召回率	平均精度	参数量/M	平均漏检率	平均误检率	推理时间/ms
YOLOv5s	0.785	0.829	7.09	0.334	0.026	18.9
YOLOv5s_AF−FPN	0.824	0.870	8.12	0.365	0.027	15.0
MobileNetV3_YOLOv5s	0.766	0.795	3.56	0.365	0.027	15.0
MobileNetV3_YOLOv5s_AF−FPN	0.810	0.841	4.59	0.297	0.023	16.5

下载: 导出CSV

表 4 多目标跟踪结果对比

Table 4 Comparison of multi-target tracking results

模型	MOTA/%	MOTP/%	漏检数	误检数	推理速度/ （帧·s⁻¹）
YOLOv5s+DeepSORT	60.5	76.5	119	57	34
MobileNetV3_YOLOv5s_ AF−FPN+DeepSORT	63.4	80.1	95	42	40

下载: 导出CSV

参考文献(20)

[1]	王国法,赵国瑞,任怀伟. 智慧煤矿与智能化开采关键核心技术分析[J]. 煤炭学报,2019,44(1):34-41. DOI: 10.13225/j.cnki.jccs.2018.5034 WANG Guofa,ZHAO Guorui,REN Huaiwei. Analysis on key technologies of intelligent coal mine and intelligent mining[J]. Journal of China Coal Society,2019,44(1):34-41. DOI: 10.13225/j.cnki.jccs.2018.5034
[2]	王国法,徐亚军,张金虎,等. 煤矿智能化开采新进展[J]. 煤炭科学技术,2021,49(1):1-10. WANG Guofa,XU Yajun,ZHANG Jinhu,et al. New development of intelligent mining in coal mines[J]. Coal Science and Technology,2021,49(1):1-10.
[3]	张渤,谢金辰,张后斌. 矿井下输送带大块物体检测[J]. 煤炭技术,2021,40(4):154-156. DOI: 10.13301/j.cnki.ct.2021.04.046 ZHANG Bo,XIE Jinchen,ZHANG Houbin. Detection of large objects in transportation belt under mine[J]. Coal Technology,2021,40(4):154-156. DOI: 10.13301/j.cnki.ct.2021.04.046
[4]	许军,吕俊杰,杨娟利,等. 基于图像处理的溜槽堆煤预警研究[J]. 煤炭技术,2017,36(12):232-234. XU Jun,LYU Junjie,YANG Juanli,et al. Research on early warning of coal chute blocking based on image processing[J]. Coal Technology,2017,36(12):232-234.
[5]	张立亚. 矿山智能视频分析与预警系统研究[J]. 工矿自动化,2017,43(11):16-20. ZHANG Liya. Research on intelligent video analysis and early warning system for mine[J]. Industry and Mine Automation,2017,43(11):16-20.
[6]	吴帅,徐勇,赵东宁. 基于深度卷积网络的目标检测综述[J]. 模式识别与人工智能,2018,31(4):335-346. DOI: 10.16451/j.cnki.issn1003-6059.201804005 WU Shuai,XU Yong,ZHAO Dongning. Survey of object detection based on deep convolutional networks[J]. Pattern Recognition and Artificial Intelligence,2018,31(4):335-346. DOI: 10.16451/j.cnki.issn1003-6059.201804005
[7]	管皓,薛向阳,安志勇. 深度学习在视频目标跟踪中的应用进展与展望[J]. 自动化学报,2016,42(6):834-847. DOI: 10.16383/j.aas.2016.c150705 GUAN Hao,XUE Xiangyang,AN Zhiyong. Advances on application of deep learning for video object tracking[J]. Acta Automatica Sinica,2016,42(6):834-847. DOI: 10.16383/j.aas.2016.c150705
[8]	罗海波,许凌云,惠斌,等. 基于深度学习的目标跟踪方法研究现状与展望[J]. 红外与激光工程,2017,46(5):14-20. LUO Haibo,XU Lingyun,HUI Bin,et al. Status and prospect of target tracking based on deep learning[J]. Infrared and Laser Engineering,2017,46(5):14-20.
[9]	南柄飞, 郭志杰, 王凯, 等. 基于视觉显著性的煤矿井下关键目标对象实时感知研究[J/OL]. 煤炭科学技术: 1-11[2022-01-12]. https://kns.cnki. net/kcms/detail/11.2402.TD.20210512.1304.004.html. NAN Bingfei, GUO Zhijie, WANG Kai, et al. Real-time method of target ROI in coal mine underground based on visual saliency [J/OL]. Coal Science and Technology: 1-11[2022-01-12]. https://kns.cnki.net/kcms/detail/11.2402.TD.20210512.1304.004.html.
[10]	杜京义,郝乐,王悦阳,等. 一种煤矿井下输煤大块物检测方法[J]. 工矿自动化,2020,46(5):63-68. DU Jingyi,HAO Le,WANG Yueyang,et al. A detection method for large blocks in underground coal transportation[J]. Industry and Mine Automation,2020,46(5):63-68.
[11]	WANG Yujing,WANG Yuanbin,DANG Langfei. Video detection of foreign objects on the surface of belt conveyor underground coal mine based on improved SSD[J]. Journal of Ambient Intelligence and Humanized Computing,2020,46(7):1-10.
[12]	胡璟皓,高妍,张红娟,等. 基于深度学习的带式输送机非煤异物识别方法[J]. 工矿自动化,2021,47(6):57-62,90. DOI: 10.13272/j.issn.1671-251x.2021020041 HU Jinghao,GAO Yan,ZHANG Hongjuan,et al. Recognition method of non-coal foreign objects of belt conveyor based on deep learning[J]. Industry and Mine Automation,2021,47(6):57-62,90. DOI: 10.13272/j.issn.1671-251x.2021020041
[13]	叶鸥,窦晓熠,付燕,等. 融合轻量级网络和双重注意力机制的煤块检测方法[J]. 工矿自动化,2021,47(12):75-80. YE Ou,DOU Xiaoyi,FU Yan,et al. Coal block detection method integrating lightweight network and dual attention mechanism[J]. Industry and Mine Automation,2021,47(12):75-80.
[14]	张伟,庄幸涛,王雪力,等. DS−YOLO:一种部署在无人机终端上的小目标实时检测算法[J]. 南京邮电大学学报(自然科学版),2021,41(1):86-98. ZHANG Wei,ZHUANG Xingtao,WANG Xueli,et al. DS-YOLO:a real-time small object detection algorithm on UAVs[J]. Journal of Nanjing University of Posts and Telecommunications ( Natural Science Edition),2021,41(1):86-98.
[15]	MITTAL S. A survey on optimized implementation of deep learning models on the NVIDIA Jetson platform[J]. Journal of Systems Architecture,2019,97:428-442. DOI: 10.1016/j.sysarc.2019.01.011
[16]	WANG Junfan,CHEN Yi,GAO Mingyu,et al. Improved YOLOv5 network for real-time multi-scale traffic sign detection[J]. IEEE Sensors Journal,2021,38(8):1724-1733.
[17]	WANG Chenyao, LIAO Hongyuan, WU Yuehua, et al. CSPNet: a new backbone that can enhance learning capability of CNN[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle, 2020: 390-391.
[18]	HE Kaiming,ZHANG Xingyu,REN Shaoqing,et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2015,37(9):1904-1916. DOI: 10.1109/TPAMI.2015.2389824
[19]	LIU Shu, QI Lu, QIN Haifang, et al. Path aggregation network for instance segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 8759-8768.
[20]	BEWLEY A, GE Z, OTT L, et al. Simple online and real time tracking[C]//2016 IEEE International Conference on Image Processing （ ICIP）, Phoenix, 2016: 3464-3468.

施引文献(4)

期刊类型引用(3)

1.	齐泽中，周俊，叶子渭，周学剑，杨浩勇. 单侧制动式转向履带底盘自主行走局部路径规划方法. 农业机械学报. 2024(06): 52-59 . 百度学术
2.	王文明. 随钻取心成套装备的研发及现场应用. 矿业安全与环保. 2023(04): 97-104 . 百度学术
3.	高启业，高明，宋丽军. 履带行走车辆行走轨迹纠偏方法研究. 中国工程机械学报. 2023(06): 527-531 . 百度学术