依托Hadoop架构的海量变压器实时监测与存储方案构建

Construction of massive transformer real-time monitoring and storage solution relying on Hadoop framework

随着智能电网的建设以及电力变压器在线监测技术的成熟,电力变压器在线监测数据呈现出体量大、类型多等特点。使用传统存储技术存储变压器在线监测数据,已不能满足实时、快速的需求。为此,设计基于Hadoop集群的变压器在线监测数据存储方案。该方案利用HBase(分布式列式数据库)具有快速实时读写数据的优势,将变压器在线监测系统采集的海量数据实时快速地存储。为能自动快速实时收集数据和避免因数据流过大造成系统崩溃,分别采用Flume(日志收集工具)和Kafka(分布式流处理平台)收集和缓存数据。以电力变压器在线监测的油色谱数据为例,验证了所提存储方案的可行性和有效性。

With the construction of smart grids and the maturity of on-line monitoring technology for power transformers,the on-line monitoring data of power transformers are characterized by large volumes and various types. Using traditional storage technology to store transformer online monitoring data can no longer meet real-time and fast requirements. Therefore,a Hadoop cluster-based transformer on-line monitoring data storage scheme is designed. This solution utilizes HBase( distributed columnar database) with the advantages of quickly reading and writing data in real time,and storing the massive data collected by the transformer on-line monitoring system quickly and in real time. In order to automatically and quickly collect data in real time and avoid system crashes due to excessive data flow,Flume( log collection tool) and Kafka( distributed stream processing platform) are utilized to collect and cache data,respectively. Finally,the oil chromatogram data of power transformer on-line monitoring is taken as an example to verify the feasibility and effectiveness of the proposed storage scheme.