Spark平台的分布式阶段自适应关联规则挖掘算法

Distributed Stage Adaptive Association Rules Mining Algorithm Based on Spark

为满足日益增长的海量数据挖掘需求,迫切需要设计一种能够在多台机器上运行的分布式关联规则挖掘算法。Apriori这种高度迭代算法在Hadoop平台上运行时每次迭代执行大量的磁盘I/O操作,大大影响并限制了算法的运行效率。本文利用Spark对分布式计算内置支持的特点,在Spark平台上设计并实现一种分布式关联规则挖掘算法,称为阶段式自适应挖掘算法(Staged Adaptive Apriori)。算法使用自适应的数据集部分处理的策略对频繁项集进行高效挖掘,在每次迭代前初步评估执行时间,并采用较为合适的方法来减少时间和空间的复杂性,是一种基于数据集性质的自适应关联规则挖掘算法。实验结果表明了算法的有效性。

In order to meet the growing demand for massive data mining,it is urgent to design a distributed association rule mining algorithm that can run on multiple machines.Apriori is a highly iterative algorithm that performs a large number of disk I/O operations per iteration when running on the Hadoop platform,greatly affecting and limiting the efficiency of the algorithm.This paper uses Spark to support the characteristics of memory distribution calculation and designs and implements a distributed association rule mining algorithm called Staged Adaptive Apriori on the Spark platform.The algorithm uses the adaptive data set partial processing strategy to efficiently mine frequent itemsets.The algorithm initially evaluates the execution time before each iteration,and adopts a more appropriate method to reduce the complexity of time and space.It is an adaptive association rule mining algorithm based on the nature of data sets.The experimental results demonstrate the effectiveness of the algorithm.