Design of a regional coal mine supervision data service platform based on big data technology

There are problems in coal mine supervision system, such as multi-level repeated uploading of networked data and data inconsistencies in the safety risk monitoring and early warning system, the lack of data storage and computing resources at all levels of safety supervision agencies, the difficulty of correlation analysis and data mining, etc. In order to solve the above problems, a regional coal mine supervision data service platform based on big data technology is designed. The platform uses Kafka distributed message queues to generate standardized Kafka data bodies from regional coal mine monitoring data, and upload them to the cloud platform Kafka cluster in batches. The platform provides monitoring real-time data consumption services through the publish-subscribe mode, which reduces network transmission overhead and avoids the adverse effects of multi-level uploading and filtering of networked data. The Spark Structured Streaming computing engine and Spark SQL are used for real-time data calculation and statistical analysis of historical data. Various data analysis and mining algorithms are integrated to provide support for data mining and prediction and early warning. HBase column storage database is used to achieve reliable storage of massive historical data. Through the Hive data warehouse associated with HBase, the platform establishes various subject data model libraries to meet the demand for multi-dimensional correlation analysis of data. The data subscription service with unified security authority authentication provides the required coal mine monitoring data, statistical analysis data and data mining results for supervision agencies at all levels, decouples the back-end data service center from the front-end supervision and monitoring business system, and provides data customization and consumer services for supervision agencies at all levels through data services so as to improve data utilization efficiency. The application results show that the platform can meet the storage, analysis and calculation and data sharing needs of regional coal mine monitoring data.