基于云计算的电力大数据分析技术与应用

Cloud-Computing Based Power Big Data Analysis Technology and Its Application

为解决电力数据分析系统在大数据时代面临的严重的性能与可伸缩性瓶颈,更好地满足生产、营销等系统的需求,分析了云计算技术的优势,提出了基于云计算的电力大数据分析系统体系结构及关键技术。基于分布式并行计算框架Hadoop和Hive,面向电力大数据特征,设计了多维索引、SQL自动翻译工具和支持数据更新的混合存储模型3项性能提升技术,实现对传统电力数据分析系统的升级优化。在浙江电力用电信息采集系统的实际部署经验表明,和传统电力数据分析系统相比,该系统以1/8的硬件成本,获得平均5倍的性能优势。证明了云计算技术能够显著提升电力大数据查询与分析性能并有效降低成本。

In order to deal with the bottlenecks of the performance and scalability of power data analysis in present big data era and meet the requirements of power production and marking systems, the advantages of cloud computing technology is firstly analyzed, then the architecture and key techniques of Hadoop-based power big data analysis system are proposed. The architecture combines the characteristics of power big data and the emerging cloud computing technology, and includes three key technologies, i.e., the distributed gridfile based multi-dimensional index, the automatic SQL to HiveQL translation tool and the hybrid data storage model, which can support data update operation. Therefore, the traditional power data analysis system can be improved and updated. The application of the system in the power information collection system in Zhejiang indicates that the performance of the system is improved by 5 times with only one-eighth of the hardware configuration as compared with the traditional system. This proves that the cloud computing technology can significantly improve the performance and reduce the cost of big data acquisition and analysis.