实时大数据处理技术在状态监测领域中的应用

Real-Time Big Data Processing Technology Application in the Field of State Monitoring

随着智能电网建设和研究的不断推进,对输变电设备状态监测的广度和深度不断加强,状态监测过程中收集的数据量呈指数级增长。然而,电力系统要求对生产、管理、运营能够实时监控,对在线监测系统的实时性要求很高。现有的大数据处理技术(如Map Reduce等适合离线大数据分析)应用于在线状态监测系统时,其性能很难保证。根据状态监测数据特点,利用Storm实时处理监测数据流,设计了数据流处理拓扑结构和消息树;利用Spark内存集群计算技术,提高状态评价和数据分析算法的性能,设计了k-means的聚类算法,实现数据的聚类划分。最后提出了结合大数据处理、实时流数据处理和内存批处理技术的状态监测数据实时分析框架。

With the development of smart grid construction and research, the power transmission and transformation equipment state monitoring are expanding in breadth and depth, in the process of state monitoring collected data is growing exponentially. However, the power system requirements on-line monitoring for production, management and operation, on-line monitoring system is high real-time demand. Existing big data processing techniques(such as Map Reduce suitable for offline data analysis) are applied to the real-time state monitoring system, the performance is very difficult to guarantee. According to characteristics of state monitoring data, using the Storm real-time processing monitoring data stream, data stream processing topology structure and the message tree are designed. Using the Spark in-memory cluster computing technology, improve the performance of state evaluation and data analysis algorithms, design the k- means clustering algorithm, realization of data clustering. At last, we designed state monitoring real-time analysis framework combination with big data processing, real-time streaming data processing and in-memory batch processing technology.