基于Hadoop生态圈的选煤数据中台设计

赵鑫; 王然风; 付翔

doi:10.13272/j.issn.1671-251x.2021040004

基于Hadoop生态圈的选煤数据中台设计

doi: 10.13272/j.issn.1671-251x.2021040004

太原理工大学矿业工程学院, 山西太原 030024

基金项目:

山西省应用基础研究计划重点自然基金项目(201901D111007)；山西省关键核心技术和共性技术研发攻关专项项目(2020XXX004)。

详细信息

作者简介:
赵鑫(1996-)，男，山西太原人，硕士研究生，研究方向为选煤智能化与大数据开发,E-mail：zhaoxin3158@163.com。

中图分类号: TD948
计量
- 文章访问数: 142
- HTML全文浏览量: 20
- PDF下载量: 16
- 被引次数: 0
出版历程
- 收稿日期: 2021-04-02
- 修回日期: 2021-12-04
- 刊出日期: 2021-12-20

Design of coal preparation data center platform based on Hadoop ecosystem

College of Mining Engineering, Taiyuan University of Technology, Taiyuan 030024, China

摘要

摘要: 针对现有选煤厂信息管理系统采用的接口不规范，导致数据重复采集，且各系统相互独立，对多源异构数据处理能力弱等问题，基于Hadoop生态圈大数据技术，提出了一种基于Hadoop生态圈的选煤数据中台设计方案。通过主数据管理系统、企业服务总线定义数据标准实现系统集成；设计归一化、相关系数矩阵和噪声异常点检测程序实现数据处理；结合D-S（Dempster-Shafer）证据理论、Hadoop与Hive数据仓库设计多源异构数据融合子系统，实现数据融合；利用Highcharts数据可视化组件实现数据交互式的可视化展示。实际应用结果表明，该数据中台实现了主数据定义标准与系统集成接口规范化，提高了选煤数据处理能力，实现了多源异构选煤数据融合共享、数据实时交互式的可视化展示。
- 选煤厂智能化 /
- Hadoop生态圈 /
- 数据中台 /
- 大数据 /
- 多源异构数据 /
- 数据融合 /
- D-S证据理论
Abstract: The existing coal preparation plant information management system uses nonstandard interface, which leads to repeated data collection, and each system is independent of each other, and the capability to process multi-source heterogeneous data is weak. In order to solve above problems, based on big data technology of Hadoop ecosystem, a coal preparation data center platform design scheme based on Hadoop ecosystem is proposed. The system integration is realized by defining data standards through master data management system and enterprise service bus. Normalization, correlation coefficient matrix and noise abnormal point detection programs are designed to realize data processing. DS (Dempster-Shafer) evidence theory, Hadoop and Hive data warehouse are combined to design multi-source heterogeneous data fusion subsystem to realize data fusion. Highcharts data visualization components are used to achieve interactive visualization of data. The practical application results show that the data center platform realizes the standardization of master data definition standard and system integration interface, improves the processing capability of coal preparation data, realizes the fusion and sharing of multi-source heterogeneous coal preparation data, and realizes the real-time interactive visualization of data.
- intelligent coal preparation plant /
- Hadoop ecosystem /
- data cente platform /
- big data /
- multi-source heterogeneous data /
- data fusion /
- D-S evidence theory

HTML全文

参考文献(15)

[1]	刘银志,赵廷钊,毛善君,等.大型煤矿集团管理智能化系统研究[J].工矿自动化,2020,46(12):25-30. LIU Yinzhi,ZHAO Tingzhao,MAO Shanjun,et al.Research on intelligent management system of large coal group[J].Industry and Mine Automation,2020,46(12):25-30.
[2]	董永胜,陈为高,侯佃平,等.智能化选煤厂研究与建议[J].工矿自动化,2021,47(增刊1):26-31. DONG Yongsheng,CHEN Weigao,HOU Dianping,et al.Research and suggestions on intelligent coal preparation plant[J].Industry and Mine Automation,2021,47(S1):26-31.
[3]	王国法,王虹,任怀伟,等.智慧煤矿2025情景目标和发展路径[J].煤炭学报,2018,43(2):295-305. WANG Guofa,WANG Hong,REN Huaiwei,et al.2025 scenarios and development path of intelligent coal mine[J].Journal of China Coal Society,2018,43(2):295-305.
[4]	疏礼春.智能煤矿数据中台架构及关键技术研究[J].工矿自动化,2021,47(6):40-44. SHU Lichun.Research on the architecture and key technologies of intelligent coal mine data middle platform[J].Industry and Mine Automation,2021,47(6):40-44.
[5]	马小平,代伟.大数据技术在煤炭工业中的研究现状与应用展望[J].工矿自动化,2018,44(1):50-54. MA Xiaoping,DAI Wei.Research status and application prospect of big data technology in coal industry[J].Industry and Mine Automation,2018,44(1):50-54.
[6]	匡亚莉.智能化选煤厂建设的内涵与框架[J].选煤技术,2018(1):85-91. KUANG Yali.The intension and framework for the construction of intelligent coal preparation plant[J].Coal Preparation Technology,2018(1):85-91.
[7]	张磊.信息自动化技术在选煤厂的应用探讨[J].煤炭工程,2018,50(增刊1):125-127. ZHANG Lei.Discussion on the application of information automation technology in coal preparation plant[J].Coal Engineering,2018,50(S1):125-127.
[8]	高晓茜,杨永胜,白龙,等.石圪台选煤厂信息化调度管理[J].洁净煤技术,2019,25(增刊1):161-163. GAO Xiaoqian,YANG Yongsheng,BAI Long,et al.Information dispatching management of Shigetai coal preparation plant[J].Clean Coal Technology,2019,25(S1):161-163.
[9]	孙小路,周春侠,张永志,等.选煤生产过程标准数据平台建设及其关键技术[J].煤炭工程,2021,53(1):38-42. SUN Xiaolu,ZHOU Chunxia,ZHANG Yongzhi,et al.Construction of standard data platform for coal preparation process and its key technologies[J].Coal Engineering,2021,53(1):38-42.
[10]	刘贤康.基于Hadoop生态圈的工业数据平台设计与研究[D].武汉:华中科技大学,2019. LIU Xiankang.Design and research of industrial data platform based on Hadoop ecosphere[D].Wuhan:Huazhong University of Science and Technology,2019.
[11]	杜小勇,卢卫,张峰.大数据管理系统的历史、现状与未来[J].软件学报,2019,30(1):127-141. DU Xiaoyong,LU Wei,ZHANG Feng.History,present,and future of big data management systems[J].Journal of Software,2019,30(1):127-141.
[12]	曹现刚,罗璇,张鑫媛,等.煤矿机电设备运行状态大数据管理平台设计[J].煤炭工程,2020,52(2):22-26. CAO Xiangang,LUO Xuan,ZHANG Xinyuan,et al.Design of big data management platform for operation status of coal mine electromechanical equipment[J].Coal Engineering,2020,52(2):22-26.
[13]	王万丽,孙超,宿国瑞.基于云平台的煤矿安全智能管控信息平台设计[J].煤炭工程,2019,51(6):52-56. WANG Wanli,SUN Chao,SU Guorui.Design of coal mine safety intelligent control information platform based on cloud platform[J].Coal Engineering,2019,51(6):52-56.
[14]	刘纪芹,史开泉.大数据分解-融合及其智能获取[J].计算机科学,2020,47(6):66-73. LIU Jiqin,SHI Kaiquan.Big data decomposition-fusion and its intelligent acquisition[J].Computer Science,2020,47(6):66-73.
[15]	陈金环,王涛,李帅,等.基于改进D-S证据理论的智能安全传感器[J].信息技术与信息化,2021(2):232-235. CHEN Jinhuan,WANG Tao,LI Shuai,et al.Intelligent security sensor based on improved D-S evidence theory[J].Information Technology and Information Technology,2021(2):232-235.