基于监控PaaS的大数据治理研究

Research on big data governance based on monitoring PaaS

为解决区域能源互联网数据质量差的问题,设计了基于监控PaaS的大数据治理框架,通过PaaS平台、DM8MPP数据库.Spark计算引擎等技术,支撑构建消息传输桥接代理、分片内存索引快照、流计算数据清洗、数据混合存储等传统ETL工具缺乏的流计算模块,实现云原生数据采集处理与数据治理技术的融合互补。本文采用Kafka消息动态代理,实现跨层、广域、无感知的消息发布/订阅;基于HashMap和智能指针构建分片内存快照,扩展实现索引快照,实现跨节点、跨容器的快速信息访问;研究流计算数据清洗技术,实现多源遥测、遥信的连续真值计算;研究混合数据存储模型,解决大数据平台多样化数据的分布式存储及访问问题。通过仿真测试结果分析,本研究提高了能源大数据接入及存储水平,为数据服务总线提供更方便的数据访问支持。

In order to solve the problems of poor data quality of regional energy internet,a big data governance framework based on monitoring PaaS is designed.Through PaaS,DM8MPP,Spark computing engine and other technologies,it supports the construction of streaming computing module,which is lack of traditional ETL-tools such as message transmission bridging agent,partitioned memory index snapshot,streaming data cleaning,data mixed storage,to realize the integration and complementarity of scada and data governance using cloud native technology.In this paper,Kafka dynamic agent is used to realize cross layer,wide area and non perception message publish/subscribe;based on HashMap and SmartPointers,partitioned memory snapshot is constructed,and index snapshot is extended to realize message access across nodes and dockers;streaming data cleaning technology is studied to realize continuous truth calculation of multi-source telemetry and remote signaling;hybrid data storage model is studied to solve the problem of distributed storage and access in big data platform.Through the analysis of simulation test results,this study improves the access and storage level of energy big data,and provides more convenient data access support for data service bus.