基于图像融合和改进CornerNet-Squeeze的煤矿井下行人检测方法

邹盛; 周李兵; 季亮; 于政乾

doi:10.13272/j.issn.1671-251x.2022070001

基于图像融合和改进CornerNet-Squeeze的煤矿井下行人检测方法

doi: 10.13272/j.issn.1671-251x.2022070001

邹盛^{1, 2},
周李兵^{1, 2},
季亮^{1, 2},
于政乾^{1, 2}

1.
中煤科工集团常州研究院有限公司, 江苏常州　213015
2.
天地(常州)自动化股份有限公司, 江苏常州　213015)

基金项目: 江苏省科技成果转化专项项目（BA2022040）；中国煤炭科工集团有限公司科技创新创业资金专项重点项目（2021-TD-ZD004）；天地科技股份有限公司科技创新创业资金专项项目（2021-TD-ZD004）；中煤科工集团常州研究院有限公司科研项目（2022TY6001）。

详细信息

作者简介:
邹盛（1993—），男，湖北麻城人，工程师，硕士，主要从事机器视觉图像算法相关工作，E-mail:695449327 @qq.com

中图分类号: TD67
计量
- 文章访问数: 230
- HTML全文浏览量: 181
- PDF下载量: 46
- 被引次数: 0
出版历程
- 收稿日期: 2022-07-01
- 修回日期: 2023-02-01
- 网络出版日期: 2022-09-19

A pedestrian target detection method for underground coal mine based on image fusion and improved CornerNet-Squeeze

ZOU Sheng^{1, 2},
ZHOU Libing^{1, 2},
JI Liang^{1, 2},
YU Zhengqian^{1, 2}

1.
CCTEG Changzhou Research Institute, Changzhou 213015, China
2.
Tiandi(Changzhou) Automation Co., Ltd., Changzhou 213015, China

摘要

摘要: 在煤矿井下无人驾驶和安防监控等领域，对行人目标的检测至关重要，但受井下光线昏暗、光照不均、背景复杂、行人目标小且密集等特殊工况环境的影响，图像中的行人目标存在边缘细节特征少、信噪比低、与背景相似度高等问题，难以有效识别遮挡多尺度下的行人目标。针对上述问题，提出了一种基于图像融合和改进CornerNet-Squeeze的煤矿井下行人目标检测方法。采用双尺度图像融合（TIF）算法将红外相机和深度相机采集的图像进行像素级融合，再进行形态学处理，减少背景干扰。在CornerNet-Squeeze网络基础上，将八度卷积（OctConv）引入沙漏型主干网络，处理图像特征中高低频信息，增强图像边缘特征，提高多尺度行人检测能力。实验结果表明：① 在深度图像、红外图像、融合图像3种数据集上，改进CornerNet-Squeeze模型在保持原算法实时性的同时，有效提升了井下行人检测精度。② 采用融合图像数据集训练的模型检测精度较红外图像和深度图像数据集训练的模型高，可见融合图像能充分发挥深度图像和红外图像的优势，有助于提高模型检测精度。③ 在不同程度遮挡和多尺度行人目标6种场景下，改进CornerNet-Squeeze训练的模型的行人漏检率最低。④ 与YOLOv4 相比，在 COCO2014 行人数据集上改进CornerNet-Squeeze算法的平均精度提高了 1.1%，检测速度提高了6.7%。⑤ 改进CornerNet-Squeeze能够有效检测出图像中远处小目标，对小目标的检测能力提升明显。
- 煤矿井下行人检测 /
- 双尺度图像融合 /
- CornerNet-Squeeze /
- 边缘增强 /
- 沙漏型主干网络 /
- 多尺度行人检测
Abstract: In unmanned driving and security monitoring in the coal mine, detecting pedestrian targets is very important. But under the influence of special working conditions such as dim light, uneven illumination, complex background, and small and dense pedestrian targets, the pedestrian targets in the image have some problems such as few edge details, low signal-to-noise ratio and high similarity with the background. It is difficult to effectively identify the pedestrian targets under multi-scale occlusion. In order to solve the above problems, a pedestrian detection method for underground coal mine based on image fusion and improved CornerNet-Squeeze is proposed. The image collected by the infrared camera and depth camera is fused at the pixel level using the two-scale image fusion (TIF) algorithm. The morphological processing is carried out for the fused imoge to reduce background interference. Based on the CornerNet-Squeeze network, octave convolution (OctConv) is introduced into the hourglass type backbone network to process the high and low frequency information of image features, so as to enhance the image edge features and improve the detection capability of multi-scale pedestrians. The experimental results show the following points. ① The improved CornerNet-Squeeze model can effectively improve the detection precision of underground pedestrian while maintaining the real-time performance of the original algorithm on the data sets of range image, infrared image and fusion image. ② The detection precision of the model trained by the fusion image dataset is higher than that of the models trained by the infrared image dataset or the depth image dataset. The result shows that the fusion image can give full play to the advantages of the depth image and the infrared image, and is helpful to improve the detection precision of the model. ③ In the six scenes of different degrees of occlusion and multi-scale pedestrian target, the model trained by the improved CornerNet-Squeeze has the lowest pedestrian misdetection rate. ④ Compared with YOLOv 4, the average accuracy of the improved CornerNet-Squeeze algorithm on the COCO2014 pedestrian dataset is improved by 1.1%, and the detection speed is improved by 6.7%. ⑤ The improved CornerNet-Squeeze can effectively detect the small target in the image. The detection capability of the small target is obviously improved.
- underground pedestrian detection /
- two-scale image fusion /
- CornerNet-Squeeze /
- edge enhancement /
- hourglass type backbone network /
- multi-scale pedestrian detection

HTML全文

图 1 CornerNet网络结构

Figure 1. CornerNet network structure

下载: 全尺寸图片幻灯片

图 2 Hourglass-52 Network网络结构

In−输入模块；Pool 池化模块；Res 残差模块；Up 上采样模块；Out 输出模块

Figure 2. Hourglass-52 Network structure

下载: 全尺寸图片幻灯片

图 3 改进CornerNet-Squeeze网络结构

Figure 3. Improved CornerNet-Squeeze network structure

下载: 全尺寸图片幻灯片

图 4 OctConv操作过程

Figure 4. OctConv operation procedure

下载: 全尺寸图片幻灯片

图 5 图像融合处理原理

Figure 5. Principle of image fusion processing

下载: 全尺寸图片幻灯片

图 6 图像融合处理过程

Figure 6. Process of image fusion

下载: 全尺寸图片幻灯片

图 7 同一数据集下验证损失值曲线

Figure 7. Validation-Loss value curve under the same data set

下载: 全尺寸图片幻灯片

图 8 3种数据的检测结果

Figure 8. Test results of three kinds of data

下载: 全尺寸图片幻灯片

表 1 不同模型的行人目标检测性能

Table 1. Pedestrian target detection performance of different models

数据集	模型	A/%	FPS/（帧·s⁻¹）
红外图像	CornerNet	71.18	24
	CornerNet-Squeeze	73.68	31
	改进CornerNet-Squeeze	78.36	31
深度图像	CornerNet	72.11	25
	CornerNet-Squeeze	74.56	30
	改进CornerNet-Squeeze	78.21	29
融合图像	CornerNet	75.76	22
	CornerNet-Squeeze	82.63	28
	改进CornerNet-Squeeze	85.36	28

下载: 导出CSV

表 2 不同背景下行人目标检测效果

Table 2. Pedestrian target detection effect in different backgrounds

测试场景	目标总数/个	漏检率/%
测试场景	目标总数/个	CornerNet	CornerNet- Squeeze	改进 CornerNet-Squeeze
轻微遮挡	200	2.82	2.33	1.81
部分遮挡	160	11.43	9.12	8.36
严重遮挡	60	52.37	48.81	43.54
大尺寸目标	180	1.55	1.46	1.22
中小尺寸目标	150	7.39	6.88	6.12
极小尺寸目标	50	40.33	35.78	31.68

下载: 导出CSV

表 3 在COCO2014 行人数据集上性能对比

Table 3. Performance comparison on the COCO2014 pedestrian dataset

算法	检测速度/ms	A/%	A^s/%	A^m/%	A^b/%
YOLOv4	30	43.2	13.2	45.4	65.6
改进CornerNet-Squeeze	32	44.3	18.1	44.1	64.3

下载: 导出CSV

参考文献(21)

[1]	王珂硕. 基于深度学习的井下行人检测方法研究[D]. 哈尔滨: 黑龙江科技大学, 2022. WANG Keshuo. Research on pedestrian detection method in mines based on deep learning[D]. Harbin: Heilongjiang University of Science and Technology, 2022.
[2]	杨清翔,吕晨,冯晨晨,等. 煤矿井下行人检测算法[J]. 工矿自动化,2020,46(1):80-84. doi: 10.13272/j.issn.1671-251x.17540 YANG Qingxiang,LYU Chen,FENG Chenchen,et al. Pedestrian detection algorithm of coal mine underground[J]. Industry and Mine Automation,2020,46(1):80-84. doi: 10.13272/j.issn.1671-251x.17540
[3]	李晓明,郎文辉,马忠磊,等. 基于图像处理的井下机车行人检测技术[J]. 煤矿机械,2017,38(4):167-170. doi: 10.13436/j.mkjx.201704059 LI Xiaoming,LANG Wenhui,MA Zhonglei,et al. Pedestrian detection technology for mine locomotive based on image processing[J]. Coal Mine Machinery,2017,38(4):167-170. doi: 10.13436/j.mkjx.201704059
[4]	REN Shaoqing,HE Kaiming,GIRSHICK R,et al. Faster R-CNN:towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2017,39(6):1137-1149.
[5]	唐聪,凌永顺,郑科栋,等. 基于深度学习的多视窗SSD目标检测方法[J]. 红外与激光工程,2018,47(1):302-310. TANG Cong,LING Yongshun,ZHENG Kedong,et al. Object detection method of multi-view SSD based on deep learning[J]. Infrared and Laser Engineering,2018,47(1):302-310.
[6]	REDMON J, DIVVALA S, GIRSHICK R, et al. You only look once: unified, real-time object detection[C]. IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, 2016: 779-788.
[7]	LAW Hei,DENG Jia. CornerNet:detecting objects as paired Keypoints[J]. International Journal of Computer Vision,2020,128(2):642-656.
[8]	李伟山,卫晨,王琳. 改进的Faster RCNN煤矿井下行人检测算法[J]. 计算机工程与应用,2019,55(4):200-207. doi: 10.3778/j.issn.1002-8331.1711-0282 LI Weishan,WEI Chen,WANG Lin. lmproved Faster RCNN approach for pedestrian detection in underground coal mine[J]. Computer Engineering and Applications,2019,55(4):200-207. doi: 10.3778/j.issn.1002-8331.1711-0282
[9]	李现国,李斌,刘宗鹏,等. 井下视频行人检测方法[J]. 工矿自动化,2020,46(2):54-58. doi: 10.13272/j.issn.1671-251x.2019060024 LI Xianguo,LI Bin,LIU Zongpeng,et al. Underground video pedestrian detection method[J]. Industry and Mine Automation,2020,46(2):54-58. doi: 10.13272/j.issn.1671-251x.2019060024
[10]	张明臻. 基于Dense-YOLO网络的井下行人检测模型[J]. 工矿自动化,2022,48(3):86-90. doi: 10.13272/j.issn.1671-251x.17861 ZHANG Mingzhen. Underground pedestrian detection model based on Dense-YOLO network[J]. Journal of Mine Automation,2022,48(3):86-90. doi: 10.13272/j.issn.1671-251x.17861
[11]	张庆庆,朱仲杰,高明,等. 基于CornerNet-Lite的输电塔与绝缘子目标识别与检测[J]. 浙江万里学院学报,2020,33(3):85-90. doi: 10.13777/j.cnki.issn1671-2250.2020.03.014 ZHANG Qingqing,ZHU Zhongjie,GAO Ming,et al. Target recognition and detection of transmission tower and insulator based on Corner Net-Lite[J]. Journal of Zhejiang Wanli University,2020,33(3):85-90. doi: 10.13777/j.cnki.issn1671-2250.2020.03.014
[12]	赵文清,孔子旭,赵振兵. 隔级融合特征金字塔与CornerNet相结合的小目标检测[J]. 智能系统学报,2021,16(1):108-116. ZHAO Wenqing,KONG Zixu,ZHAO Zhenbing. Small target detection based on a combination of feature pyramid and CornerNet[J]. CAAI Transactions on Intelligent Systems,2021,16(1):108-116.
[13]	刘宇航,马健霄,王羽尘,等. 基于改进CornerNet-Lite的林区行人检测算法[J]. 林业工程学报,2021,6(4):154-156. LIU Yuhang,MA Jianxiao,WANG Yuchen,et al. Forest-pedestrian detection algorithm based on improved CornerNet-Lite[J]. Journal of Forestry Engineering,2021,6(4):154-156.
[14]	刘子威,邓春华,刘静. 基于非对称沙漏网络结构的目标检测算法[J]. 计算机应用,2020,40(12):3526-3533. LIU Ziwei,DENG Chunhua,LIU Jing. Object detection algorithm based on asymmetric hourglass network structure[J]. Journal of Computer Applications,2020,40(12):3526-3533.
[15]	吴佳豪,周凤,李亮亮. 基于堆叠沙漏网络的人体姿态估计[J]. 计算机系统应用,2021,30(10):295-300. WU Jiahao,ZHOU Feng,LI Liangliang. Human pose estimation based on stacked hourglass network[J]. Computer Systems & Applications,2021,30(10):295-300.
[16]	江洪,宋勇,隋国成,等. 智能选矸机器人系统的研究与应用[J]. 选煤技术,2020(4):81-87. doi: 10.16447/j.cnki.cpt.2020.04.019 JIANG Hong,SONG Yong,SUI Guocheng,et al. Study and application of the intelligent robotic gangue picking system[J]. Coal Preparation Technology,2020(4):81-87. doi: 10.16447/j.cnki.cpt.2020.04.019
[17]	李坤伦,魏泽发,宋焕生. 基于SqueezeNet卷积神经网络的车辆颜色识别[J]. 长安大学学报(自然科学版),2020,40(4):109-116. LI Kunlun,WEI Zefa,SONG Huansheng. Vehicle color recognition based on SqueezeNet[J]. Journal of Chang'an University(Natural Science Edition),2020,40(4):109-116.
[18]	王继霄,李阳,王家宝,等. 基于SqueezeNet的轻量级图像融合方法[J]. 计算机应用,2020,40(3):837-841. WANG Jixiao,LI yang,WANG Jiabao. Light-weight image fusion method based on SqueezeNet[J]. Journal of Computer Applications,2020,40(3):837-841.
[19]	蒙双,陈乐庚,肖晨晨. 基于改进OctConv的车道线检测算法研究[J]. 计算机仿真,2021,38(5):142-145,218. MENG Shuang,CHEN Legeng,XIAO Chenchen. Researchon lane detection algorithm based on improved OctConv[J]. Computer Simulation,2021,38(5):142-145,218.
[20]	高原,陈爱斌,周国雄,等. 基于OctConv的DCNN在遥感图像场景分类中的应用[J]. 电子测量与仪器学报,2020,34(1):61-67. GAO Yuan,CHEN Aibin,ZHOU Guoxiong,et al. Application of DCNN based on OctConv in scene classification of remote sensing images[J]. Journal of Electronic Measurement and Instrumentation,2020,34(1):61-67.
[21]	BAVIRISETI DP,DHULI R. Two-scale image fusion of visible and infrared images using saliency detection[J]. Infrared Physics & Technology,2016,76:52-64.