基于大数据技术的区域煤矿监管数据服务平台设计

Design of a regional coal mine supervision data service platform based on big data technology

针对煤矿监管体系中安全风险监测预警系统联网数据多级重复上传、数据不一致现象,各级安全监管机构数据存储计算资源缺乏、关联分析和数据挖掘困难等问题,设计了基于大数据技术的区域煤矿监管数据服务平台。该平台采用Kafka分布式消息队列,将区域内煤矿监测监控数据生成标准化Kafka数据体,批量上传至云平台Kafka集群;通过发布-订阅模式提供监测监控实时数据消费服务,减轻了网络传输开销,避免了联网数据多级上传和过滤带来的不利影响;采用Spark Structured Streaming计算引擎和Spark SQL进行数据实时计算和历史数据统计分析,通过集成的各类数据分析挖掘算法,为数据挖掘和预测预警提供支持;采用HBase列存储数据库实现海量历史数据的可靠存储,通过与HBase相关联的Hive数据仓库建立各类主题数据模型库,满足数据的多维关联分析需求;通过统一安全权限认证的数据订阅服务为各级监管机构提供所需的煤矿监测监控数据、统计分析数据和数据挖掘结果,将后台数据服务中心与前端监管监察业务系统解耦,通过数据服务为各级监管机构提供数据定制和消费服务,提高了数据利用效率。应用结果表明,平台可满足区域煤矿监测监控数据的存储、分析计算和数据共享需求。

There are problems in coal mine supervision system,such as multi-level repeated uploading of networked data and data inconsistencies in the safety risk monitoring and early warning system,the lack of data storage and computing resources at all levels of safety supervision agencies,the difficulty of correlation analysis and data mining,etc.In order to solve the above problems,a regional coal mine supervision data service platform based on big data technology is designed.The platform uses Kafka distributed message queues to generate standardized Kafka data bodies from regional coal mine monitoring data,and upload them to the cloud platform Kafka cluster in batches.The platform provides monitoring real-time data consumption services through the publish-subscribe mode,which reduces network transmission overhead and avoids the adverse effects of multi-level uploading and filtering of networked data.The Spark Structured Streaming computing engine and Spark SQL are used for real-time data calculation and statistical analysis of historical data.Various data analysis and mining algorithms are integrated to provide support for data mining and prediction and early warning.HBase column storage database is used to achieve reliable storage of massive historical data.Through the Hive data warehouse associated with HBase,the platform establishes various subject data model libraries to meet the demand for multi-dimensional correlation analysis of data.The data subscription service with unified security authority authentication provides the required coal mine monitoring data,statistical analysis data and data mining results for supervision agencies at all levels,decouples the back-end data service center from the front-end supervision and monitoring business system,and provides data customization and consumer services for supervision agencies at all levels through data services so as to improve data utilization efficiency.The application results show that the platform can meet the storage,analysis and calculation and data sharing needs of regional coal mine monitoring data.