基于雷达与视觉融合的双模态煤矿井下环境感知技术

杨志方

doi:10.13272/j.issn.1671-251x.2023080073

基于雷达与视觉融合的双模态煤矿井下环境感知技术

doi: 10.13272/j.issn.1671-251x.2023080073

杨志方^{1, 2, 3,}

1.
煤炭科学技术研究院有限公司，北京　100013
2.
煤矿应急避险技术装备工程研究中心，北京　100013
3.
北京市煤矿安全工程技术研究中心，北京　100013

基金项目: 国家自然科学基金青年基金项目（42201386）；天地科技股份有限公司科技创新创业资金专项（2023-TD-ZD005-005，2022-2-TD-ZD001，2022-TD-ZD001）。

详细信息

作者简介:
杨志方（1994—），男，河南洛阳人，硕士，研究方向为矿山人工智能技术，E-mail：hnezzsf@163.com

中图分类号: TD67
计量
- 文章访问数: 280
- HTML全文浏览量: 91
- PDF下载量: 56
- 被引次数: 0
出版历程
- 收稿日期: 2023-08-19
- 修回日期: 2023-11-11
- 网络出版日期: 2023-11-23

Bimodal environment perception technology for underground coal mine based on radar and visual fusion

YANG Zhifang^{1, 2, 3
,}

1.
CCTEG China Coal Research Institute, Beijing 100013, China
2.
Engineering Research Center for Technology Equipment of Emergency Refuge in Coal Mine, Beijing 100013, China
3.
Beijing Engineering and Technology Research Center of Mine Safety, Beijing 100013, China

摘要

摘要: 环境感知是煤矿巡检机器人、视觉测量系统等场景应用的关键技术。单模态环境感知技术对煤矿井下复杂环境的感知能力较差。提出了雷达与视觉双模态空间融合方法，通过激光雷达和摄像仪之间的坐标转换来实现二者采集信息的融合，从而提高环境感知能力。为了更好地提取目标特征信息，提出了双模态融合环境感知网络架构技术路线：摄像仪和雷达采集的环境信息经雷达与视觉双模态空间融合方法进行融合处理，多模态特征融合网络模块提取融合信息中的目标特征，多任务处理网络模块采用不同的任务头处理目标特征信息，完成目标检测、图像分割、目标分类等环境感知任务。采用YOLOv5s目标检测算法搭建双模态特征提取网络模块进行实验，结果表明：基于雷达与视觉融合的双模态煤矿井下环境感知技术对井下巷道环境下行人检测的成功率较视觉、雷达感知分别提升15%，10%，对车道线、标志牌等各类目标分割的平均精度均值较视觉感知均提高10%以上，有效提升了煤矿井下环境感知能力，可为煤矿道路环境感知、视觉测量系统、无人矿车导航系统、矿井搜救机器人等应用场景提供技术支持。
- 煤矿井下环境感知 /
- 雷达感知 /
- 视觉感知 /
- 双模态融合感知 /
- 图像分割 /
- 目标分类识别
Abstract: Environmental perception is a key technology for scenario applications such as coal mine inspection robots and visual measurement systems. The single modal environmental perception technology has poor perception capability for complex environments in underground coal mines. A bimodal space fusion method for radar and vision has been proposed. The modal achieves the fusion of information collected by LiDAR and camera through coordinate conversion, thereby improving environmental perception capability. In order to better extract object feature information, a bimodal fusion environment perception network architecture technology route is proposed. The environmental information collected by the camera and radar is fused and processed by the radar and visual bimodal space fusion method. The multimodal feature fusion network module extracts object features from the fused information. The multitask processing network module uses different task heads to process object feature information, completing environmental perception tasks such as object detection, image segmentation, and object classification. The experiment is conducted using the YOLOv5s object detection algorithm to build a bimodal feature extraction network module. The results show that the success rate of the bimodal environment perception technology for underground coal mine based on radar and visual fusion for personnel detection in underground roadway environments is improved by 15% and 10% compared to visual and radar perception, respectively. The mean average precision of segmentation for various types of objects such as lane lines and signs are improved by more than 10% compared to visual perception. It effectively improves the perception capability of underground environment in coal mines, providing technical support for application scenarios such as coal mine road environment perception, visual measurement systems, unmanned mining vehicle navigation systems, and mine search and rescue robots.
- perception of underground environment in coal mines /
- radar perception /
- visual perception /
- bimodal fusion perception /
- image segmentation /
- object classification and recognition

HTML全文

图 1 坐标系转换关系

Figure 1. Conversion relationship among coordinate systems

下载: 全尺寸图片幻灯片

图 2 雷达坐标系与世界坐标系的转换关系

Figure 2. Conversion relationship between radar coordinate system and world coordinate system

下载: 全尺寸图片幻灯片

图 3 相机成像模型

Figure 3. Camera imaging model

下载: 全尺寸图片幻灯片

图 4 像素坐标系与图像坐标系

Figure 4. Pixel coordinate system and image coordinate system

下载: 全尺寸图片幻灯片

图 5 双模态融合环境感知技术网络架构

Figure 5. Network architecture of bimodal fusion environment perception technology

下载: 全尺寸图片幻灯片

图 6 双模态特征提取网络模块架构

Figure 6. Architecture of bimodal feature extraction network module

下载: 全尺寸图片幻灯片

图 7 固定点位场景应用

Figure 7. Fixed position scenario application

下载: 全尺寸图片幻灯片

图 8 移动场景应用

Figure 8. Mobile scenario application

下载: 全尺寸图片幻灯片

图 9 煤矿井下行人感知结果

Figure 9. Personnel perception results in underground coal mine

下载: 全尺寸图片幻灯片

图 10 煤矿井下目标分割结果

Figure 10. Object segmentation results in underground coal mine

下载: 全尺寸图片幻灯片

CPU	i9−9820X
GPU	NVIDIA RTX 2080 Ti GPU×2
操作系统	Ubuntu 20.04
架构	PyTorch
实验总轮数	100
批数大小	4
初始学习率	0.001
模型优化器	AdamW
权重衰减项	1×10⁻⁴

下载: 导出CSV

表 1 模型轻量化效果

Table 1. Model lightweighting effects

模型	模型大小/MiB	平均精度/%	平均帧率/ （帧·s⁻¹）	每秒浮点运算次数/10⁶
轻量化前	256	87.7	28.7	53.17
轻量化后	52.6	82.6	49.8	30.23

下载: 导出CSV

表 2 单模态与双模态检测结果对比

Table 2. Comparison between of unimodal and bimodal detection results

感知方式	目标总数/个	检测数/个	漏检数/个	成功率/%
视觉	60	45	15	75
雷达	60	48	12	80
融合	60	54	6	90

下载: 导出CSV

表 3 煤矿井下各类目标分割结果的评价指标

Table 3. Evaluation indexes of segmentation results of various types of object in underground coal mine %

目标	纯图像			双模态空间融合图像
目标	mIoU	召回率	mAP	mIoU	召回率	mAP
背景	83.21	75.74	99.22	85.62	89.61	99.78
车辆	77.48	81.62	79.82	79.40	84.50	88.92
路面	79.27	79.80	72.31	82.12	91.30	82.38
车道线	74.21	67.90	68.35	80.21	91.56	77.62
标志牌	74.01	78.33	60.54	78.88	90.37	79.85
行人	84.21	83.44	66.95	90.25	95.65	83.44

下载: 导出CSV

参考文献(29)

[1]	胡青松,孟春蕾,李世银,等. 矿井无人驾驶环境感知技术研究现状及展望[J]. 工矿自动化,2023,49(6):128-140. HU Qingsong,MENG Chunlei,LI Shiyin,et al. Research status and prospects of perception technology for unmanned mining vehicle driving environment[J]. Journal of Mine Automation,2023,49(6):128-140.
[2]	XING Zhizhong,ZHAO Shuanfeng,GUO Wei,et al. Identifying balls feature in a large-scale laser point cloud of a coal mining environment by a multiscale dynamic graph convolution neural network[J]. ACS Omega,2022,7(6):4892-4907. doi: 10.1021/acsomega.1c05473
[3]	王国法,赵国瑞,任怀伟. 智慧煤矿与智能化开采关键核心技术分析[J]. 煤炭学报,2019,44(1):34-41. WANG Guofa,ZHAO Guorui,REN Huaiwei. Analysis on key technologies of intelligent coal mine and intelligent mining[J]. Journal of China Coal Society,2019,44(1):34-41.
[4]	鲍久圣,章全利,葛世荣,等. 煤矿井下无人化辅助运输系统关键基础研究及应用实践[J]. 煤炭学报,2023,48(2):1085-1098. doi: 10.13225/j.cnki.jccs.2022.1600 BAO Jiusheng,ZHANG Quanli,GE Shirong,et al. Basic research and application practice of unmanned auxiliary transportation system in coal mine[J]. Journal of China Coal Society,2023,48(2):1085-1098. doi: 10.13225/j.cnki.jccs.2022.1600
[5]	LI Xiaohu,WAN Shaoke,LIU Shijie,et al. Bearing fault diagnosis method based on attention mechanism and multilayer fusion network[J]. ISA Transactions,2022,128:550-564. doi: 10.1016/j.isatra.2021.11.020
[6]	ZHANG Di. Interoperability technology of sports health monitoring equipment based on multi-sensor information fusion[J]. Eurasip Journal on Advances in Signal Processing,2021,2021(1):1-18. doi: 10.1186/s13634-020-00710-6
[7]	蔡峰,孔令华,程志恒. 大型煤炭企业煤矿智能化建设进展、问题和对策研究[J]. 中国煤炭,2023,49(6):14-18. doi: 10.3969/j.issn.1006-530X.2023.06.003 CAI Feng,KONG Linghua,CHENG Zhiheng. Research on the progress,problems,and countermeasures of intelligent construction in large coal enterprises' coal mines[J]. China Coal,2023,49(6):14-18. doi: 10.3969/j.issn.1006-530X.2023.06.003
[8]	张鹏鹏,俞阿龙,孙诗裕,等. 多传感器数据融合在矿井安全监测中的应用[J]. 工矿自动化,2015,41(12):5-8. ZHANG Pengpeng,YU Along,SUN Shiyu,et al. Application of multi-sensor data fusion in mine safety monitoring[J]. Industry and Mine Automation,2015,41(12):5-8.
[9]	张静. 基于多传感器融合的露天矿山障碍物检测方法研究[D]. 长沙:湖南大学,2022. ZHANG Jing. The obstacle detection method based on multi-sensor fusion in open-pit mines[D]. Changsha:Hunan University,2022.
[10]	胡荣华,安冬,史梦圆,等. 智能矿用机器人研究现状及发展趋势[J]. 黄金,2023,44(9):59-68. doi: 10.11792/hj20230910 HU Ronghua,AN Dong,SHI Mengyuan,et al. Research situation and development tendency of intelligent mining robots[J]. Gold,2023,44(9):59-68. doi: 10.11792/hj20230910
[11]	杨静宜,赵莉娅,梁月肖. 基于机器视觉的矿用巡检机器人环境感知研究[J]. 煤炭技术,2023,42(9):227-229. doi: 10.13301/j.cnki.ct.2023.09.047 YANG Jingyi,ZHAO Liya,LIANG Yuexiao. Study on mine patrol robot environment perception based on machine vision[J]. Coal Technology,2023,42(9):227-229. doi: 10.13301/j.cnki.ct.2023.09.047
[12]	党相卫,秦斐,卜祥玺,等. 一种面向智能驾驶的毫米波雷达与激光雷达融合的鲁棒感知算法[J]. 雷达学报,2021,10(4):622-631. doi: 10.12000/JR21036 DANG Xiangwei,QIN Fei,BU Xiangxi,et al. A Robust perception algorithm based on a radar and LiDAR for intelligent driving[J]. Journal of Radars,2021,10(4):622-631. doi: 10.12000/JR21036
[13]	江良玉. 基于激光雷达与毫米波雷达的矿区路侧感知算法[J]. 控制与信息技术,2022(5):75-79. JIANG Liangyu. Roadside perception system based on LiDAR and millimeter-wave radar fusion in mining area[J]. Control and Information Technology,2022(5):75-79.
[14]	张海波. 基于视觉与激光雷达融合的煤岩识别技术研究[D]. 徐州:中国矿业大学,2022. ZHANG Haibo. Research on coal-rock recognition technology based on fusion of vision and lidar[D]. Xuzhou:China University of Mining and Technology,2022.
[15]	袁晓明,郝明锐. 煤矿辅助运输机器人关键技术研究[J]. 工矿自动化,2020,46(8):8-14. YUAN Xiaoming,HAO Mingrui. Research on key technologies of coal mine auxiliary transportation robot[J]. Industry and Mine Automation,2020,46(8):8-14.
[16]	杨春雨,张鑫. 煤矿机器人环境感知与路径规划关键技术[J]. 煤炭学报,2022,47(7):2844-2872. YANG Chunyu,ZHANG Xin. Key technologies of coal mine robots for environment perception and path planning[J]. Journal of China Coal Society,2022,47(7):2844-2872.
[17]	陈少杰,朱振才,张永合,等. 基于3D特征点的激光雷达与立体视觉配准方法[J]. 激光与光电子学进展,2020,57(3):58-65. CHEN Shaojie,ZHU Zhencai,ZHANG Yonghe,et al. Extrinsic calibration for lidar and stereo vision using 3D feature points[J]. Laser & Optoelectronics Progress,2020,57(3):58-65.
[18]	张赛赛,于红绯. 基于多维动态卷积的激光雷达与相机外参标定方法[J/OL]. 激光与光电子学进展:1-12[2023-08-03]. http://kns.cnki.net/kcms/detail/31.1690.TN.20230920.1803.070.html. ZHANG Saisai,YU Hongfei. LiDAR and camera external parameter calibration method based on multi-dimensional dynamic convolution[J/OL]. Laser & Optoelectronics Progress:1-12[2023-08-03]. http://kns.cnki.net/kcms/detail/31.1690.TN.20230920.1803.070.html.
[19]	PHILION J,FIDLER S. Lift,splat,shoot:encoding images from arbitrary camera rigs by implicitly unprojecting to 3D[C]. Computer Vision-ECCV,Glasgow,2020:194-210.
[20]	LI Zhiqi,WANG Wenhai,LI Hongyang,et al. Bevformer:learning bird's-eye-view representation from multi-camera images via spatiotemporal transformers[C]. European Conference on Computer Vision,Tel Aviv,2022:1-18.
[21]	许志,李敬兆,张传江,等. 轻量化CNN及其在煤矿智能视频监控中的应用[J]. 工矿自动化,2020,46(12):13-19. XU Zhi,LI Jingzhao,ZHANG Chuanjiang,et al. Lightweight CNN and its application in coal mine intelligent video surveillance[J]. Industry and Mine Automation,2020,46(12):13-19.
[22]	魏东,王忠宾,司垒,等. 采煤机作业区域人员精确检测方法研究[J]. 工矿自动化,2022,48(2):19-28. WEI Dong,WANG Zhongbin,SI Lei,et al. Research on precise detection method of personnel in shearer operation area[J]. Industry and Mine Automation,2022,48(2):19-28.
[23]	饶中钰,吴景涛,李明. 煤矸石图像分类方法[J]. 工矿自动化,2020,46(3):69-73. RAO Zhongyu,WU Jingtao,LI Ming. Coal-gangue image classification method[J]. Industry and Mine Automation,2020,46(3):69-73.
[24]	杜青,杨仕教,郭钦鹏,等. 地下矿山作业人员佩戴安全帽智能检测方法[J]. 工矿自动化,2023,49(7):134-140. DU Qing,YANG Shijiao,GUO Qinpeng,et al. Intelligent detection method of working personnel wearing safety helmets in underground mine[J]. Journal of Mine Automation,2023,49(7):134-140.
[25]	余乐文. 地下矿山非结构环境地图创建与自主探索方法研究[D]. 北京:北京科技大学,2022. YU Lewen. Research on methods of mapping and autonomous exploration in underground mine unstructured environment[D]. Beijing:University of Science and Technology Beijing,2022.
[26]	RONNEBERGER O,FISCHER P,BROX T. U-net:convolutional networks for biomedical image segmentation[C]. International Conference on Medical Image Computing and Computer Assisted Intervention,Munich,2015:234-241.
[27]	沈科,季亮,张袁浩,等. 基于改进YOLOv5s模型的煤矸目标检测[J]. 工矿自动化,2021,47(11):107-111,118. SHEN Ke,JI Liang,ZHANG Yuanhao,et al. Research on coal and gangue detection algorithm based on improved YOLOv5s model[J]. Industry and Mine Automation,2021,47(11):107-111.
[28]	张旭辉,闫建星,张超,等. 基于改进YOLOv5s+DeepSORT的煤块行为异常识别[J]. 工矿自动化,2022,48(6):77-86,117. ZHANG Xuhui,YAN Jianxing,ZHANG Chao,et al. Coal block abnormal behavior identification based on improved YOLOv5s + DeepSORT[J]. Journal of Mine Automation,2022,48(6):77-86,117.
[29]	WANG C Y,LIAO H Y M,WU Y H,et al. CSPNet:a new backbone that can enhance learning capability of CNN[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops,Seattle,2020:390-391.