Volume 49 Issue 11
Nov.  2023
Turn off MathJax
Article Contents
Article Navigation >  Journal of Mine Automation  > 2023  >  49(11): 67-75
YANG Zhifang. Bimodal environment perception technology for underground coal mine based on radar and visual fusion[J]. Journal of Mine Automation，2023，49（11）：67-75.  doi: 10.13272/j.issn.1671-251x.2023080073
Citation: YANG Zhifang. Bimodal environment perception technology for underground coal mine based on radar and visual fusion[J]. Journal of Mine Automation，2023，49（11）：67-75.  doi: 10.13272/j.issn.1671-251x.2023080073
shu

Bimodal environment perception technology for underground coal mine based on radar and visual fusion

doi: 10.13272/j.issn.1671-251x.2023080073
  • 1.

    CCTEG China Coal Research Institute, Beijing 100013, China

  • 2.

    Engineering Research Center for Technology Equipment of Emergency Refuge in Coal Mine, Beijing 100013, China

  • 3.

    Beijing Engineering and Technology Research Center of Mine Safety, Beijing 100013, China

  • Received Date: 2023-08-19
  • Rev Recd Date: 2023-11-11
    • Available Online: 2023-11-23
    Abstract
  • Environmental perception is a key technology for scenario applications such as coal mine inspection robots and visual measurement systems. The single modal environmental perception technology has poor perception capability for complex environments in underground coal mines. A bimodal space fusion method for radar and vision has been proposed. The modal achieves the fusion of information collected by LiDAR and camera through coordinate conversion, thereby improving environmental perception capability. In order to better extract object feature information, a bimodal fusion environment perception network architecture technology route is proposed. The environmental information collected by the camera and radar is fused and processed by the radar and visual bimodal space fusion method. The multimodal feature fusion network module extracts object features from the fused information. The multitask processing network module uses different task heads to process object feature information, completing environmental perception tasks such as object detection, image segmentation, and object classification. The experiment is conducted using the YOLOv5s object detection algorithm to build a bimodal feature extraction network module. The results show that the success rate of the bimodal environment perception technology for underground coal mine based on radar and visual fusion for personnel detection in underground roadway environments is improved by 15% and 10% compared to visual and radar perception, respectively. The mean average precision of segmentation for various types of objects such as lane lines and signs are improved by more than 10% compared to visual perception. It effectively improves the perception capability of underground environment in coal mines, providing technical support for application scenarios such as coal mine road environment perception, visual measurement systems, unmanned mining vehicle navigation systems, and mine search and rescue robots.

     

    • FullText(HTML)
  • loading
    • References(29)
  • [1]
    胡青松,孟春蕾,李世银,等. 矿井无人驾驶环境感知技术研究现状及展望[J]. 工矿自动化,2023,49(6):128-140.

    HU Qingsong,MENG Chunlei,LI Shiyin,et al. Research status and prospects of perception technology for unmanned mining vehicle driving environment[J]. Journal of Mine Automation,2023,49(6):128-140.
    [2]
    XING Zhizhong,ZHAO Shuanfeng,GUO Wei,et al. Identifying balls feature in a large-scale laser point cloud of a coal mining environment by a multiscale dynamic graph convolution neural network[J]. ACS Omega,2022,7(6):4892-4907. doi: 10.1021/acsomega.1c05473
    [3]
    王国法,赵国瑞,任怀伟. 智慧煤矿与智能化开采关键核心技术分析[J]. 煤炭学报,2019,44(1):34-41.

    WANG Guofa,ZHAO Guorui,REN Huaiwei. Analysis on key technologies of intelligent coal mine and intelligent mining[J]. Journal of China Coal Society,2019,44(1):34-41.
    [4]
    鲍久圣,章全利,葛世荣,等. 煤矿井下无人化辅助运输系统关键基础研究及应用实践[J]. 煤炭学报,2023,48(2):1085-1098. doi: 10.13225/j.cnki.jccs.2022.1600

    BAO Jiusheng,ZHANG Quanli,GE Shirong,et al. Basic research and application practice of unmanned auxiliary transportation system in coal mine[J]. Journal of China Coal Society,2023,48(2):1085-1098. doi: 10.13225/j.cnki.jccs.2022.1600
    [5]
    LI Xiaohu,WAN Shaoke,LIU Shijie,et al. Bearing fault diagnosis method based on attention mechanism and multilayer fusion network[J]. ISA Transactions,2022,128:550-564. doi: 10.1016/j.isatra.2021.11.020
    [6]
    ZHANG Di. Interoperability technology of sports health monitoring equipment based on multi-sensor information fusion[J]. Eurasip Journal on Advances in Signal Processing,2021,2021(1):1-18. doi: 10.1186/s13634-020-00710-6
    [7]
    蔡峰,孔令华,程志恒. 大型煤炭企业煤矿智能化建设进展、问题和对策研究[J]. 中国煤炭,2023,49(6):14-18. doi: 10.3969/j.issn.1006-530X.2023.06.003

    CAI Feng,KONG Linghua,CHENG Zhiheng. Research on the progress,problems,and countermeasures of intelligent construction in large coal enterprises' coal mines[J]. China Coal,2023,49(6):14-18. doi: 10.3969/j.issn.1006-530X.2023.06.003
    [8]
    张鹏鹏,俞阿龙,孙诗裕,等. 多传感器数据融合在矿井安全监测中的应用[J]. 工矿自动化,2015,41(12):5-8.

    ZHANG Pengpeng,YU Along,SUN Shiyu,et al. Application of multi-sensor data fusion in mine safety monitoring[J]. Industry and Mine Automation,2015,41(12):5-8.
    [9]
    张静. 基于多传感器融合的露天矿山障碍物检测方法研究[D]. 长沙:湖南大学,2022.

    ZHANG Jing. The obstacle detection method based on multi-sensor fusion in open-pit mines[D]. Changsha:Hunan University,2022.
    [10]
    胡荣华,安冬,史梦圆,等. 智能矿用机器人研究现状及发展趋势[J]. 黄金,2023,44(9):59-68. doi: 10.11792/hj20230910

    HU Ronghua,AN Dong,SHI Mengyuan,et al. Research situation and development tendency of intelligent mining robots[J]. Gold,2023,44(9):59-68. doi: 10.11792/hj20230910
    [11]
    杨静宜,赵莉娅,梁月肖. 基于机器视觉的矿用巡检机器人环境感知研究[J]. 煤炭技术,2023,42(9):227-229. doi: 10.13301/j.cnki.ct.2023.09.047

    YANG Jingyi,ZHAO Liya,LIANG Yuexiao. Study on mine patrol robot environment perception based on machine vision[J]. Coal Technology,2023,42(9):227-229. doi: 10.13301/j.cnki.ct.2023.09.047
    [12]
    党相卫,秦斐,卜祥玺,等. 一种面向智能驾驶的毫米波雷达与激光雷达融合的鲁棒感知算法[J]. 雷达学报,2021,10(4):622-631. doi: 10.12000/JR21036

    DANG Xiangwei,QIN Fei,BU Xiangxi,et al. A Robust perception algorithm based on a radar and LiDAR for intelligent driving[J]. Journal of Radars,2021,10(4):622-631. doi: 10.12000/JR21036
    [13]
    江良玉. 基于激光雷达与毫米波雷达的矿区路侧感知算法[J]. 控制与信息技术,2022(5):75-79.

    JIANG Liangyu. Roadside perception system based on LiDAR and millimeter-wave radar fusion in mining area[J]. Control and Information Technology,2022(5):75-79.
    [14]
    张海波. 基于视觉与激光雷达融合的煤岩识别技术研究[D]. 徐州:中国矿业大学,2022.

    ZHANG Haibo. Research on coal-rock recognition technology based on fusion of vision and lidar[D]. Xuzhou:China University of Mining and Technology,2022.
    [15]
    袁晓明,郝明锐. 煤矿辅助运输机器人关键技术研究[J]. 工矿自动化,2020,46(8):8-14.

    YUAN Xiaoming,HAO Mingrui. Research on key technologies of coal mine auxiliary transportation robot[J]. Industry and Mine Automation,2020,46(8):8-14.
    [16]
    杨春雨,张鑫. 煤矿机器人环境感知与路径规划关键技术[J]. 煤炭学报,2022,47(7):2844-2872.

    YANG Chunyu,ZHANG Xin. Key technologies of coal mine robots for environment perception and path planning[J]. Journal of China Coal Society,2022,47(7):2844-2872.
    [17]
    陈少杰,朱振才,张永合,等. 基于3D特征点的激光雷达与立体视觉配准方法[J]. 激光与光电子学进展,2020,57(3):58-65.

    CHEN Shaojie,ZHU Zhencai,ZHANG Yonghe,et al. Extrinsic calibration for lidar and stereo vision using 3D feature points[J]. Laser & Optoelectronics Progress,2020,57(3):58-65.
    [18]
    张赛赛,于红绯. 基于多维动态卷积的激光雷达与相机外参标定方法[J/OL]. 激光与光电子学进展:1-12[2023-08-03]. http://kns.cnki.net/kcms/detail/31.1690.TN.20230920.1803.070.html.

    ZHANG Saisai,YU Hongfei. LiDAR and camera external parameter calibration method based on multi-dimensional dynamic convolution[J/OL]. Laser & Optoelectronics Progress:1-12[2023-08-03]. http://kns.cnki.net/kcms/detail/31.1690.TN.20230920.1803.070.html.
    [19]
    PHILION J,FIDLER S. Lift,splat,shoot:encoding images from arbitrary camera rigs by implicitly unprojecting to 3D[C]. Computer Vision-ECCV,Glasgow,2020:194-210.
    [20]
    LI Zhiqi,WANG Wenhai,LI Hongyang,et al. Bevformer:learning bird's-eye-view representation from multi-camera images via spatiotemporal transformers[C]. European Conference on Computer Vision,Tel Aviv,2022:1-18.
    [21]
    许志,李敬兆,张传江,等. 轻量化CNN及其在煤矿智能视频监控中的应用[J]. 工矿自动化,2020,46(12):13-19.

    XU Zhi,LI Jingzhao,ZHANG Chuanjiang,et al. Lightweight CNN and its application in coal mine intelligent video surveillance[J]. Industry and Mine Automation,2020,46(12):13-19.
    [22]
    魏东,王忠宾,司垒,等. 采煤机作业区域人员精确检测方法研究[J]. 工矿自动化,2022,48(2):19-28.

    WEI Dong,WANG Zhongbin,SI Lei,et al. Research on precise detection method of personnel in shearer operation area[J]. Industry and Mine Automation,2022,48(2):19-28.
    [23]
    饶中钰,吴景涛,李明. 煤矸石图像分类方法[J]. 工矿自动化,2020,46(3):69-73.

    RAO Zhongyu,WU Jingtao,LI Ming. Coal-gangue image classification method[J]. Industry and Mine Automation,2020,46(3):69-73.
    [24]
    杜青,杨仕教,郭钦鹏,等. 地下矿山作业人员佩戴安全帽智能检测方法[J]. 工矿自动化,2023,49(7):134-140.

    DU Qing,YANG Shijiao,GUO Qinpeng,et al. Intelligent detection method of working personnel wearing safety helmets in underground mine[J]. Journal of Mine Automation,2023,49(7):134-140.
    [25]
    余乐文. 地下矿山非结构环境地图创建与自主探索方法研究[D]. 北京:北京科技大学,2022.

    YU Lewen. Research on methods of mapping and autonomous exploration in underground mine unstructured environment[D]. Beijing:University of Science and Technology Beijing,2022.
    [26]
    RONNEBERGER O,FISCHER P,BROX T. U-net:convolutional networks for biomedical image segmentation[C]. International Conference on Medical Image Computing and Computer Assisted Intervention,Munich,2015:234-241.
    [27]
    沈科,季亮,张袁浩,等. 基于改进YOLOv5s模型的煤矸目标检测[J]. 工矿自动化,2021,47(11):107-111,118.

    SHEN Ke,JI Liang,ZHANG Yuanhao,et al. Research on coal and gangue detection algorithm based on improved YOLOv5s model[J]. Industry and Mine Automation,2021,47(11):107-111.
    [28]
    张旭辉,闫建星,张超,等. 基于改进YOLOv5s+DeepSORT的煤块行为异常识别[J]. 工矿自动化,2022,48(6):77-86,117.

    ZHANG Xuhui,YAN Jianxing,ZHANG Chao,et al. Coal block abnormal behavior identification based on improved YOLOv5s + DeepSORT[J]. Journal of Mine Automation,2022,48(6):77-86,117.
    [29]
    WANG C Y,LIAO H Y M,WU Y H,et al. CSPNet:a new backbone that can enhance learning capability of CNN[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops,Seattle,2020:390-391.
    • Relative Articles
    • Supplements(0)
    • Cited By
  • 加载中

Catalog

    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Figures(10)  / Tables(4)

    Get Citation
    PDF
    XML

    Article Metrics

    Article views (307) PDF downloads(62) Cited by()
    Proportional views
    Related

    苏ICP备 05016349号-2

    Supported by: Beijing Renhe Information Technology Co. Ltd.  Visits：    

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return