Massive data mining and analysis platform design for fully mechanized working face
-
摘要: 当前综采工作面海量数据采集的实时性和完整性差、异常数据清洗耗时大、数据挖掘时延大，导致综采数据利用率低，无法辅助管理层实时下发决策指令。针对上述问题，设计了一种综采工作面海量数据挖掘分析平台。该平台由数据源层、数据采集存储层、数据挖掘层和前端应用层组成。数据源层由工作面各类硬件设备提供原始数据；数据采集存储层使用OPC UA网关实时采集井下传感器监测信息，再通过MQTT协议和RESTful接口将数据存入InfluxDB存储引擎；数据挖掘层利用Hive数据引擎和Yarn资源管理器筛选数据采集过程中受工作现场干扰形成的异常数据，解决因网络延时导致的数据局部采集顺序紊乱问题，并利用Spark分布式挖掘引擎挖掘工作面设备群海量工况数据的潜在价值，提高数据挖掘模型的运行速度；前端应用层利用可视化组件与后端数据库关联，再通过AJAX技术与后端数据实时交互，实现模型挖掘结果和各类监测数据的可视化展示。测试结果表明，该平台能够充分保证数据采集的实时性与完整性，清洗效率较单机MySQL查询引擎提升5倍，挖掘效率较单机Python挖掘引擎提升4倍。Abstract: The current real-time and integrity of massive data acquisition in fully mechanized working faces are poor. The abnormal data cleaning takes a long time. The data mining delays are large. This leads to low utilization rate of fully mechanized working data and incapability to assist management in issuing decision-making instructions in real-time. In order to solve the above problems, a massive data mining and analysis platform for fully mechanized working faces is designed. The platform consists of a data source layer, a data acquisition and storage layer, a data mining layer, and a front-end application layer. The data source layer is provided with raw data by various hardware devices on the working surface. The data acquisition and storage layer uses the OPC UA gateway to collect real-time monitoring information from underground sensors, and then stores the data in the InfluxDB storage engine through the MQTT protocol and RESTful interface. The data mining layer uses the Hive data engine and Yarn resource manager to filter out abnormal data caused by workplace interference during the data acquisition process. It solves the problem of local data acquisition order disorder caused by network latency. The Spark distributed mining engine is used to explore the potential value of massive working condition data in the working face device group, improving the running speed of the data mining model. The front-end application layer utilizes visual components to associate with the back-end database. It interacts with the back-end data in real-time through AJAX technology to achieve visual display of model mining results and various monitoring data. The test results show that the platform can fully ensure the real-time and integrity of data acquisition. The cleaning efficiency is 5 times better than a standalone MySQL query engine and the mining efficiency is 4 times better than a standalone Python mining engine.
-
Key words:
- fully mechanized working face /
- massive data /
- data mining /
- data acquisition /
- data storage /
- data cleaning /
- data visualization
-
图 1 综采工作面海量数据挖掘分析平台总体架构
Figure 1. Overall architecture of massive data mining and analysis platform for fully mechanized working face
图 2 海量数据采集存储技术实现流程
Figure 2. Massive data acquisition and storage technology implementation process
-
[1] 王国法,赵国瑞,任怀伟. 智慧煤矿与智能化开采关键核心技术分析[J]. 煤炭学报,2019,44(1):34-41.WANG Guofa,ZHAO Guorui,REN Huaiwei. Analysis on key technologies of intelligent coal mine and intelligent mining[J]. Journal of China Coal Society,2019,44(1):34-41. [2] LIU Xianglan. Digital construction of coal mine big data for different platforms based on life cycle[C]. IEEE 2nd International Conference on Big Data Analysis, Beijing, 2017: 456-459. [3] 李福兴,李璐爔. 面向煤炭开采的大数据处理平台构建关键技术[J]. 煤炭学报,2019,44(增刊1):362-369.LI Fuxing,LI Luxi. Key technologies of big data processing platform construction for coal mining[J]. Journal of China Coal Society,2019,44(S1):362-369. [4] 杜毅博,赵国瑞,巩师鑫. 大数据关键技术在智能化煤矿中的应用与发展[J]. 煤炭科学技术,2020,48(7):177-185.DU Yibo,ZHAO Guorui,GONG Shixin. Study on big data platform architecture of intelligent coal mine and key technologies of data processing[J]. Coal Science and Technology,2020,48(7):177-185. [5] 王国法,王虹,任怀伟,等. 智慧煤矿2025情景目标和发展路径[J]. 煤炭学报,2018,43(2):295-305. doi: 10.13225/j.cnki.jccs.2018.0152WANG Guofa,WANG Hong,REN Huaiwei,et al. 2025 scenarios and development path of intelligent coal mine[J]. Journal of China Coal Society,2018,43(2):295-305. doi: 10.13225/j.cnki.jccs.2018.0152 [6] 崔卫锋,田野,李旭,等. 煤矿综采工作面智能服务大数据决策平台[J]. 煤矿机械,2022,43(10):177-195.CUI Weifeng,TIAN Ye,LI Xu,et al. Intelligent service big data decision-making platform for fully mechanized mining face in coal mine[J]. Coal Mine Machinery,2022,43(10):177-195. [7] 谭章禄,王美君. 智慧矿山数据治理概念内涵、发展目标与关键技术[J]. 工矿自动化,2022,48(5):6-14. doi: 10.13272/j.issn.1671-251x.2021120090TAN Zhanglu,WANG Meijun. Research on the concept connotation,development goal and key technologies of data governance for smart mine[J]. Journal of Mine Automation,2022,48(5):6-14. doi: 10.13272/j.issn.1671-251x.2021120090 [8] 葛世荣. 煤矿智采工作面概念及系统架构研究[J]. 工矿自动化,2020,46(4):1-9.GE Shirong. Research on concept and system architecture of smart mining workface in coal mine[J]. Industry and Mine Automation,2020,46(4):1-9. [9] 李首滨,李森,张守祥,等. 综采工作面智能感知与智能控制关键技术与应用[J]. 煤炭科学技术,2021,49(4):28-39.LI Shoubin,LI Sen,ZHANG Shouxiang,et al. Key technology and application of intelligent perception and intelligent control in fully mechanized mining face[J]. Coal Science and Technology,2021,49(4):28-39. [10] 苏杰,王新坤. 寸草塔煤矿综采工作面智能化建设关键技术研究与应用[J]. 煤炭科学技术,2022,50(增刊1):250-256.SU Jie,WANG Xinkun. Research and application of key technologies for intelligent construction of fully mechanized mining face in Cuncaota Coal Mine[J]. Coal Science and Technology,2022,50(S1):250-256. [11] 贺海涛. 综采工作面智能化开采系统关键技术[J]. 煤炭科学技术,2021,49(增刊1):8-15.HE Haitao. Key technology of intelligent mining system in fully-mechanized mining face[J]. Coal Science and Technology,2021,49(S1):8-15. [12] 徐亚军,张坤,李丁一,等. 超前支架自适应支护理论与应用[J]. 煤炭学报,2020,45(10):3615-3624.XU Yajun,ZHANG Kun,LI Dingyi,et al. Theory and application of self-adaptive support for advanced powered support[J]. Journal of China Coal Society,2020,45(10):3615-3624. [13] 付翔,王然风. 工作面供液系统与液压支架协同自适应控制模型设计[J]. 采矿与岩层控制工程学报,2020,2(3):90-98.FU Xiang,WANG Ranfeng. Cooperative self-adaptive control model of fluid feeding system and hydraulic supports in working face[J]. Journal of Mining and Strata Control Engineering,2020,2(3):90-98. [14] 宫文峰, 张美玲, 陈辉. 基于深度学习的旋转机械大数据智能故障诊断方法[J/OL]. 计算机集成制造系统: 1-21[2023-04-21]. http://kns.cnki.net/kcms/detail/11.5946.TP.20220612.0910.004.html.GONG Wenfeng, ZHANG Meiling, CHEN Hui. Intelligent fault diagnosis method based on deep learning of rotating machinery under big data[J/OL]. Computer Integrated Manufacturing Systems: 1-21[2023-04-21]. http://kns.cnki.net/kcms/detail/11.5946.TP.20220612.0910.004.html. [15] 丁恩杰,俞啸,廖玉波,等. 基于物联网的矿山机械设备状态智能感知与诊断[J]. 煤炭学报,2020,45(6):2308-2319. doi: 10.13225/j.cnki.jccs.zn20.0340DING Enjie,YU Xiao,LIAO Yubo,et al. Key technology of mine equipment state perception and online diagnosis under Internet of things[J]. Journal of China Coal Society,2020,45(6):2308-2319. doi: 10.13225/j.cnki.jccs.zn20.0340 [16] 雷亚国,杨彬,杜兆钧,等. 大数据下机械装备故障的深度迁移诊断方法[J]. 机械工程学报,2019,55(7):1-8. doi: 10.3901/JME.2019.07.001LEI Yaguo,YANG Bin,DU Zhaojun,et al. Deep transfer diagnosis method for machinery in big data era[J]. Journal of Mechanical Engineering,2019,55(7):1-8. doi: 10.3901/JME.2019.07.001 [17] 谢嘉成,王学文,杨兆建. 基于数字孪生的综采工作面生产系统设计与运行模式[J]. 计算机集成制造系统,2019,25(6):1381-1391.XIE Jiacheng,WANG Xuewen,YANG Zhaojian. Design and operation mode of production system of fully mechanized coal mining face based on digital twin theory[J]. Computer Integrated Manufacturing Systems,2019,25(6):1381-1391. [18] 张科学,徐兰欣,李旭,等. 透明工作面智能化开采大数据分析决策方法及系统研究[J]. 煤炭科学技术,2022,50(2):252-262.ZHANG Kexue,XU Lanxin,LI Xu,et al. Research on big data analysis and decision system of intelligent mining in transparent working face[J]. Coal Science and Technology,2022,50(2):252-262. [19] 李重重,刘清,刘军锋,等. 面向综采工作面的自动化软件设计与应用[J]. 工矿自动化,2023,49(3):124-130.LI Zhongzhong,LIU Qing,LIU Junfeng,et al. Automation software design and application for fully mechanized working face[J]. Journal of Mine Automation,2023,49(3):124-130. [20] 李佳,徐胜超. 基于云计算的智能电网大数据处理平台[J]. 计算机工程与设计,2018,39(10):3073-3079.LI Jia,XU Shengchao. Smart power system big data processing platform in cloud environments[J]. Computer Engineering and Design,2018,39(10):3073-3079. [21] 王万良,张兆娟,高楠,等. 基于人工智能技术的大数据分析方法研究进展[J]. 计算机集成制造系统,2019,25(3):529-547.WANG Wanliang,ZHANG Zhaojuan,GAO Nan,et al. Progress of big data analytics methods based on artificial intelligence technology[J]. Computer Integrated Manufacturing Systems,2019,25(3):529-547. [22] JIANG Dingde,WANG Yuqing,LYU Zhihan,et al. Big data analysis based network behavior insight of cellular networks for industry 4.0 applications[J]. IEEE Transactions on Industrial Informatics,2019,16(2):1310-1320. [23] ZAHARIA M,XIN R S,WENDELL P,et al. Apache spark:a unified engine for big data processing[J]. Communications of the Acm,2016,59(11):56-65. doi: 10.1145/2934664 [24] 罗广恒. 基于Django和MySQL的网络化测试数据查询系统研究[J]. 智能物联技术,2019,51(2):15-21,31.LUO Guangheng. Research on measure data networking query system based on Django and MySQL[J]. Technology of IoT & AI,2019,51(2):15-21,31. [25] 张锦涛,付翔,王然风,等. 智采工作面中部液压支架集群自动化后人工调控决策模型[J]. 工矿自动化,2022,48(10):20-25.ZHANG Jintao,FU Xiang,WANG Ranfeng,et al. Manual regulation and control decision model of middle hydraulic support cluster automation in the intelligent working face[J]. Journal of Mine Automation,2022,48(10):20-25. -
下载:
点击查看大图
图(8)
计量
- 文章访问数: 1032
- HTML全文浏览量: 80
- PDF下载量: 83
- 被引次数: 0
