配用电大数据多源集成及存储优化方法

Multi-source Integration and Storage Optimization Method for Big Data of Power Distribution and Utilization

面对体量大、类型多、增长快的配用电大数据,如何利用大数据技术提升配用电相关业务的广度、深度和精度成为电力行业新的机遇和挑战。为解决配用电大数据多源集成和高效存储两方面核心问题,根据配用电大数据的组成及特征,通过生成标准化元数据并构建相应数据字典的方法,实现了多源配用电数据规范化集成;在数据集成的基础上,基于Hadoop平台进行大数据存储优化方法研究,提出考虑配用电数据关联性的哈希分桶存储算法,实现了相关联数据的集中存储,从而提升后期数据查询及处理的效率并在数据存储优化的基础上,实现基于Map Reduce的多源配用电大数据并行关联查询。通过在Hadoop集群平台上进行测试表明,经过哈希分桶存储优化后的多源数据并行关联查询相比传统Hadoop方法查询时间显著缩短。

In the face of massive,heterogeneous and fast growing big data of power distribution and utilization,how to apply big data technology to improve the breadth,depth,and accuracy of power distribution and utilization business becomes a new opportunity and challenge in power industry.In order to solve the two major problems of big data with the multi-source integration and efficient storage,according to the composition and characteristics of big data of power distribution and utilization,we adopt standardized metadata and corresponding data dictionaries to realize the standardized integration of multi-source data of power distribution and utilization.On the basis of data integration,the optimization method of big data storage is studied based on the Hadoop platform.A Hash bucket algorithm considering data correlation is proposed.The algorithm realizes the centralized storage of related data,so as to enhance the efficiency of data query and processing.On the basis of data storage optimization,the parallel association query for multi-source big data of power distribution and utilization based on Map Reduce is realized.Tests on a Hadoop platform show that,after optimization of hash bucket storage,the time of the multi-source data parallel association query is significantly shortened than traditional Hadoop method.