ODPS平台下的电力设备监测大数据存储与并行处理方法

Storage and Parallel Processing of Big Data of Power Equipment Condition Monitoring on ODPS Platform

计算性能是制约电力大数据应用(基于大数据的故障诊断、预测等)的关键问题。利用分布式存储、并行计算加速此类数据密集型应用是目前较有效的手段。尝试利用阿里云开放数据处理服务(ODPS)存储并加速电力设备监测大数据分析过程。以变压器局部放电(PD)数据相位图谱分析(PRPD)为例,提出了适合高采样率、时序性强的局部放电信号数据存储方法。采用ODPS扩展MapReduce模型(MR2)设计了"Map-Reduce-Reduce"方式的PD信号宏观特征提取方法,提出了并行化PRPD分析算法(ODPS-PRPD),实现了大量PD信号的并行基本参数提取、统计特征计算与放电类型识别。在实验室中构造了4种放电模型并采集了大量PD信号,分别在ODPS平台上和实验室自建的Hadoop平台上进行了性能评估和成本分析。实验分析和结果表明,ODPS-PRPD将大量的中间过程数据(PD谱图数据等)一直保存在内存中,相比自建Hadoop MapReduce平台性能明显提升,并在数据可靠性、服务可用性以及成本方面具有明显优势。

Computing performance is one of the key issues existing in the applications of big power data,such as fault diagnosis and prediction. Distributed storage and parallel computing are currently as the effective measures to accelerate the data-intensive applications. This paper describes an open distributed processing service（ ODPS） from Ali Cloud,is used to store and accelerate the analytic process of monitoring big data about electrical equipment. Taking the phase resolved partial discharge（ PRPD） processing of a partial discharge（ PD）signal as example,a method for storing the signal with high sampling rate and time series data,and extracting the feature of the signal through the extended MapReduce model（ MR2） of ODPS is proposed in this paper. The paralleled PRPD procedure（ ODPS-PRPD） implements amounts of PD signals parallel basic parameters calculation and discharge type recognition,statistics features. To verify the effectiveness of the proposed method, a large number of partial discharge signals of four types from laboratory tests are respectively analyzed on ODPS and Hadoop. Because ODPS-PRPD stores the large amounts of middle data in the primary memory, its computing procedure is much faster. The results show that ODPS-PRPD has obviously better performance in data reliabltity,service anailabilty and cost than that of Hadoop.