基于Spark的大电网广域时空序列分析平台构建

Platform Building for Wide-Area Spatiotemporal Sequences Analysis of Large-Scale Power Grid Based on Spark

为了适应能源互联网发展趋势及日益复杂的运行环境,亟需依托大数据技术,提升能源互联网多源大数据的挖掘深度及应用效率。首先,针对大电网广域时空序列数据,阐述了Spark在分布式计算中的优势,阐明大数据平台建设目标,设计了基于Spark的电力大数据平台架构,并对平台各个层次进行详细的论述。其次,描述了Spark针对电网时空序列数据的处理过程。最后,在搭建的Spark和Hadoop实验环境基础上,对典型聚类算法进行性能对比测试,验证了Spark相对于Hadoop的MapReduce计算模型数据处理的优势,为下一步研究工作奠定了基础。

To address the energy internet trends and increasingly complex operating environment,we need to enhance the mining depth and utilization capability of energy internet multi-source data relying on big data technology. First,in the view of the wide-area spatiotemporal sequences data of large power grid,this paper expounds the Spark＇s advantages in distributed computing and the goal of big data platform,designs the big data platform architecture of power grid based on Spark,and describes each level of the platform in detail. Secondly,this paper describes the Spark＇s advantage in processing the spatiotemporal sequences data. Finally,on the basis of Spark and Hadoop experiment environment,this paper carries out typical clustering algorithm to compare the performance between Spark and Hadoop. The results verifies that Spark has a great advantage in data processing comparing with Hadoop MapReduce,which lays the foundation for the next step research.