基于多特征融合时差网络的带式输送机区域违规行为识别

马天; 姜梅; 杨嘉怡; 张杰慧; 丁旭涵

doi:10.13272/j.issn.1671-251x.2023080108

基于多特征融合时差网络的带式输送机区域违规行为识别

doi: 10.13272/j.issn.1671-251x.2023080108

马天^1,,
姜梅^1, ,,
杨嘉怡¹,
张杰慧¹,
丁旭涵²

1.
西安科技大学计算机科学与技术学院，陕西西安　710054
2.
西安科技大学安全科学与工程学院，陕西西安　710054

基金项目: 国家重点研发计划项目（2021YFB4000905）；国家自然科学基金项目（62101432，62102309）；陕西省自然科学基础研究计划项目（2022JM-508）。

详细信息

作者简介:
马天（1982—），男，河南商丘人，副教授，博士，研究方向为图形图像处理、数据可视化，E-mail：matian@xust.edu.cn

通讯作者:
姜梅（1997—），女，陕西安康人，硕士研究生，研究方向为图像处理，E-mail: 2451989925@qq.com。

中图分类号: TD634
计量
- 文章访问数: 94
- HTML全文浏览量: 26
- PDF下载量: 19
- 被引次数: 0
出版历程
- 收稿日期: 2023-08-28
- 修回日期: 2024-07-22
- 网络出版日期: 2024-08-01

Recognition of violations in belt conveyor area based on multi-feature fusion for time-difference network

1.
College of Computer Science and Technology, Xi'an University of Science and Technology, Xi'an 710054, China
2.
College of Safety Science and Engineering, Xi'an University of Science and Technology, Xi'an 710054, China

摘要

摘要: 现有的煤矿井下带式输送机区域违规行为（如攀爬、跨越、倚靠带式输送机等）识别方法对特征提取不充分、难以考虑到行为时间差异，导致违规行为识别准确率不高。针对该问题，基于ResNet50模型，提出了一种基于多特征融合时差网络（MFFTDN）的带式输送机区域违规行为识别方法，将多特征融合和时间差分进行结合，对不同时间段的行为进行多特征融合。首先在原始模型ResNet50的第2和第3阶段引入短期多特征融合(STMFF)模块，将来自多个连续帧的时间和特征拼接在一起，再对融合后的特征进行时间差分计算，即相邻帧的特征差值，以在短期内捕捉局部动作变化。然后在原始模型ResNet50的第4阶段引入长期多特征融合（LTMFF)模块，将来自连续帧的短期多特征拼接在一起，再对相邻时间点的特征进行时间差分计算，以获取行为的长期多特征。最后将融合后的特征进行分类，输出识别结果。实验结果表明：① 该方法的平均精度和准确率较原始模型ResNet50分别提高了8.18%和8.47%，说明同时引入STMFF和LTMFF模块能够有效提取到不同时间段的多特征信息。② 该方法在自建煤矿井下带式输送机区域违规行为数据集上的准确率为89.62%，平均精度为89.30%，模型的参数量为197.2 ×10⁶。③ Grad−CAM热力图显示，该方法能够更有效地关注到违规行为的关键区域，精确捕捉到井下带式输送机区域的违规行为。
- 带式输送机 /
- 不安全行为 /
- 违规行为识别 /
- 短期多特征融合 /
- 长期多特征融合 /
- 多特征融合时差网络 /
- 时间差分
Abstract: The existing methods for recognizing violations in the underground belt conveyor area of coal mines suffer from insufficient feature extraction and difficulty in considering time differences in behavior. It results in low accuracy in recognizing violations (such as climbing, crossing, leaning, etc.) in the area. In order to solve the above problems, a belt conveyor area violation recognition method based on multi-feature fusion for time-difference network (MFFTDN) is proposed using the ResNet50 model. The method combines multi-feature fusion with time difference to perform multi-feature fusion on behaviors in different time periods. Firstly, the short-term multi-feature fusion (STMFF) module is introduced in the second and third stages of the original ResNet50 model, which concatenates the time and features from multiple consecutive frames together. Then the method performs time-difference calculation on the fused features, that is, the feature difference between adjacent frames, to capture local action changes in a short period of time. Secondly, in the fourth stage of the original ResNet50 model, a long- term multi-feature fusion (LTMFF) module is introduced to concatenate short-term multi-features from consecutive frames, and perform time-difference calculations on features from adjacent time points to obtain long-term multi- feature of behavior. Finally, the method classifies the fused features and outputs the recognition results. The experimental results show the following points. ① The average accuracy and precision of the MFFTDN based belt conveyor area violation recognition method have increased by 8.18% and 8.47% respectively compared to the original model ResNet50. It indicates that the simultaneous use of STMFF and LTMFF modules can effectively extract multi-feature information from different time periods. ② The accuracy of this recognition method on the self built dataset of violations in the underground belt conveyor area of coal mines is 89.62%, with an average precision of 89.3% and a model parameter size of 197.2×10⁶. ③ The Grad CAM heatmap shows that this recognition method can more effectively focus on key areas of violations and more accurately capture violations in the underground belt area of the mine.
- belt conveyor area /
- unsafe behavior /
- recognition of violations /
- short-term multi-feature fusion /
- long- term multi-feature fusion /
- multi- feature fusion for time difference network /
- time-difference

HTML全文

图 1 MFFTDN结构

Figure 1. Structure of multi-feature fusion for time-difference network(MFFTDN)

下载: 全尺寸图片幻灯片

图 2 STMFF模块结构

Figure 2. Short-term multi-feature fusion(STMFF）module

下载: 全尺寸图片幻灯片

图 3 LTMFF模块结构

Figure 3. Long-term multi-feature fusion (LTMFF)module structure

下载: 全尺寸图片幻灯片

图 4 部分矿工行为原始数据集

Figure 4. Original dataset of some miners' behavior

下载: 全尺寸图片幻灯片

图 5 不同模型对不同行为的识别结果

Figure 5. Recognition results of different models for different behaviors

下载: 全尺寸图片幻灯片

表 1 模块消融实验结果

Table 1. Module ablation experiment table

STMFF	LTMFF	mean_acc/%	top1_acc/%	params/10⁶
×	×	81.12	81.15	186.0
√	×	85.19	85.38	197.5
×	√	88.10	88.30	197.8
√	√	89.30	89.62	197.2

下载: 导出CSV

表 2 各行为识别方法对比结果

Table 2. Comparison results of various behavior recognition methods

方法	mean_acc/%	top1_acc/%	params/10⁶
C3D	89.23	88.74	598.3
SlowFast	79.58	79.23	266.2
SlowOnly	88.46	89.33	253.6
TimesFormer	54.58	55.02	657.3
TPN	68.08	68.51	703.8
TSM	73.36	73.95	186.2
本文方法	89.30	89.62	197.2

下载: 导出CSV

参考文献(21)

[1]	张培森,李复兴,朱慧聪,等. 2008—2020年煤矿事故统计分析及防范对策[J]. 矿业安全与环保,2022,49(1):128-134. ZHANG Peisen,LI Fuxing,ZHU Huicong,et al. Statistical analysis and prevention countermeasures of coal mine accidents from 2008 to 2020[J]. Mining Safety & Environmental Protection,2022,49(1):128-134.
[2]	赵小虎,黄程龙. 基于Kinect的矿井人员违规行为识别算法研究[J]. 湖南大学学报(自然科学版),2020,47(4):92-98. ZHAO Xiaohu,HUANG Chenglong. Research on identification algorithm of mine person's violation behavior based on Kinect[J]. Journal of Hunan University(Natural Sciences),2020,47(4):92-98.
[3]	PIENAAR S W,MALEKIAN R. Human activity recognition using visual object detection[C]. IEEE 2nd Wireless Africa Conference, Pretoria,2019:1-5.
[4]	梁晨阳,华钢. 基于信道状态信息的井下人员行为识别方法研究[J]. 煤炭技术,2022,41(11):182-186. LIANG Chenyang,HUA Gang. Research on identification method of underground personnel behavior based on channel state information[J]. Coal Technology,2022,41(11):182-186.
[5]	ZHANG Xin,ZHU Yan,DENG Li,et al. A slowfast behavior recognition algorithm incorporating motion saliency[C]. 2nd International Conference on Computer Graphics,Artificial Intelligence,and Data Processing,Guangzhou,2022. DOI: 10.1117/12.2674969.
[6]	王璇,吴佳奇,阳康,等. 煤矿井下人体姿态检测方法[J]. 工矿自动化,2022,48(5):79-84. WANG Xuan,WU Jiaqi,YANG Kang,et al. Human posture detection method in coal mine[J]. Journal of Mine Automation,2022,48(5):79-84.
[7]	党伟超,张泽杰,白尚旺,等. 基于改进双流法的井下配电室巡检行为识别[J]. 工矿自动化,2020,46(4):75-800. DANG Weichao,ZHANG Zejie,BAI Shangwang,et al. Inspection behavior recognition of underground power distribution room based on improved two-stream CNN method[J]. Industry and Mine Automation,2020,46(4):75-80.
[8]	LIU Xiaoyang,LIU Jinqiang,ZHENG Haolin. Gait recognition method of coal mine personnel based on two-stream neural network[J]. Journal of Mining Science and Technology,2021,6(2):218-227.
[9]	刘斌,侯宇辉,王延辉. 基于井下轨迹数据的煤矿人员违规行为识别[J]. 煤炭与化工,2021,44(10):82-85. LIU Bin,HOU Yuhui,WANG Yanhui. Recognition of illegal behavior of coal mine personnel based on underground trajectory data[J]. Coal and Chemical Industry,2021,44(10):82-85.
[10]	WANG Zheng, LIU Yan,DUAN Siyuan,et al. An efficient detection of non-standard miner behavior using improved YOLOv8[J]. Computers and Electrical Engineering,2023,112. DOI: 10.1016/J.COMPELECENG.2023.109021.
[11]	仝泽友,丁恩杰. 矿井皮带区矿工违规行为识别方法[J]. 河南科技大学学报(自然科学版),2020,41(2):40-46,6-7. TONG Zeyou,DING Enjie. Identification method of miner violation behavior in mine belt area[J]. Journal of Henan University of Science and Technology(Natural Science),2020,41(2):40-46,6-7.
[12]	李善华,肖涛,李肖利,等. 基于DRCA−GCN的矿工动作识别模型[J]. 工矿自动化,2023,49(4):99-105,112. LI Shanhua,XIAO Tao,LI Xiaoli,et al. Miner action recognition model based on DRCA-GCN[J]. Journal of Mine Automation,2023,49(4):99-105,112.
[13]	陈天,闫雨寒,徐达伟,等. 基于改进双流算法的矿工行为识别方法研究[J]. 河南科技大学学报(自然科学版),2021,42(4):47-53,7. CHEN Tian,YAN Yuhan,XU Dawei,et al. Research on miner behavior recognition method based on improved two-stream algorithm[J]. Journal of Henan University of Science and Technology(Natural Science),2021,42(4):47-53,7.
[14]	WU Wenhao,HE Dongliang,LIN Tianwei,et al. MVFNet:multi-view fusion network for efficient video recognition[C]. AAAI Conference on Artificial Intelligence,Vancouver,2021. DOI: 10.48550/arXiv.2012.06977.
[15]	XIE Zhao,CHEN Jiansong,WU Kewei,et al. Global temporal difference network for action recognition[J]. IEEE Transactions on Multimedia,2023,25:7594-7606. doi: 10.1109/TMM.2022.3224327
[16]	TRAN D,BOURDEV L,FERGUS R,et al. Learning spatiotemporal features with 3D convolutional networks[C]. IEEE International Conference on Computer Vision,Santiago,2015:4489-4497.
[17]	FEICHTENHOFER C,FAN Haoqi,MALIK J,et al. SlowFast networks for video recognition[C]. IEEE/CVF International Conference on Computer Vision,Seoul,2019:6201-6210.
[18]	LIN Ji,GAN Chuang,HAN Song. TSM:temporal shift module for efficient video understanding[C]. IEEE/CVF International Conference on Computer Vision,Seoul,2019:7082-7092.
[19]	WANG Limin,XIONG Yuanjun,WANG Zhe,et al. Temporal segment networks:towards good practices for deep action recognition[C]. Computer Vision and Pattern Recognition,Amsterdam,2016:20-36.
[20]	BERTASIUS G,WANG Heng,TORRESANI L. Is space-time attention all you need for video understanding?[C]. International Conference on Machine Learning,Vienna,2021. DOI: 10.48550/arXiv.2102.05095.
[21]	YANG Ceyuan,XU Yinghao,SHI Jianping,et al. Temporal pyramid network for action recognition[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition,Seattle,2020:588-597.