泄漏电流数据的Spark-KNN并行模式识别方法

Parallel Pattern Recognition of Leak Current Data Using Spark-KNN

随着智能电网的快速发展,电网设备状态监测数据呈指数级增长,逐渐构成电网设备状态监测大数据。传统计算架构已无法满足计算性能需求。结合Spark大数据处理技术和阿里云E-MapReduce云计算平台,提出电网设备状态监测大数据并行模式识别方法,旨在提升电网设备在线监测系统对短时间内骤增的报警监测数据快速批量分析的能力。设计了基于Spark的并行化k最近邻分类算法(k-Nearest Neighbor,KNN)Spark-KNN,实现了海量绝缘子泄漏电流数据的并行模式识别。实验结果表明,Spark-KNN的平均性能是Hadoop MapReduce实现的2.97倍,获得了最高8.8倍的加速比,更适合执行电力设备监测大数据的实时处理任务。

With the rapid development of smart grid, the status monitoring data of power grid equipment increase exponentially and gradually form the big data. Traditional computing architectures are no longer to meet the demand of computing performance. This paper explores how Spark and Cloud computing can accelerate performance of missive insulator leak current data pattern recognition. The Parallel KNN （k-Nearest Neighbor） algorithm is designed and implemented by using Spark and Aliyun E-MapReduce cloud computing platform. The results from experiments show that the performance of Spark-KNN is 2.97 times of MapReduce-KNN and gains acceleration of 8.8 times. The experimental results confirm that Spark is more suitable for real time data processing tasks than MapReduce.