基于Spark的火电大数据挖掘方法的研究

The Research of Big Data Mining of Thermal Power Method Based on Spark

传统数据挖掘在处理火电大数据时,普遍存在计算瓶颈。针对此问题,提出了基于Spark的火电大数据挖掘方法。该方法根据机组实际运行特点,对火电大数据进行稳态工况判定和基于外部约束的工况划分,并在Spark计算框架下,引入了分布式的理念,使用基于Spark的K-means算法对火电大数据进行离散化,并使用基于Spark的FP-growth算法对火电大数据进行关联规则分析,挖掘出各工况的强关联规则,进而得到符合优化目标的参数所达到过的最优值。该方法应用到安徽某电厂300MW机组,对该机组某月10天的运行数据进行挖掘。仿真结果表明,该方法能够有效地对火电大数据进行数据挖掘,且在数据量大时。该方法与传统的数据挖掘相比计算效率具有明显优势。

There are computational bottlenecks in the traditional data mining when dealing with big data of thermal power.For this problem,a big data mining method of thermal power based on Spark is presented in this paper.According to the characteristics of the actual operation of the unit,this method determines the steady-state conditions of big data of thermal power and divides the working conditions based on external constraints,and under the framework of Spark,the concept of distributed is introduced.K-means algorithm based on Spark is used to discretize the big data of thermal power.And FP-growth algorithm based on Spark is used to analyze the association rules of thermal power big data so as to mine the strong association rules of each working condition and then get the optimal value of the parameters that meet the optimization goals.This method is applied to a 300MW unit in a power plant in Anhui Province,and mines the operation data of the unit for 10 days in a month.The results of simulation show that this method can.effectively mine big data of thermal power,and has the advantage of computational efficiency compared with the traditional data mining when the data volume is large.