分布式并行关联规则挖掘算法研究

RESEARCH ON DISTRIBUTED PARALLEL ASSOCIATION RULE MINING

关联规则挖掘算法FP-Growth虽然效率比Apriori要快一个数量级,但存在频繁模式树可能过大而内存无法容纳和数据挖掘过程串行处理等两大缺点。提出一种分布式并行关联规则挖掘算法,该算法针对分布式应用数据架构,不需要产生全局FPtree,避免全局FP-tree可能过大而内存无法容纳的问题,算法在各个主要步骤上都实现了并行处理。算法测试结果和分析表明,与传统的关联规则挖掘算法FP-Growth相比,该算法通过多节点分布式并行处理显著提高了执行效率和处理能力。

In association rule mining, though the FP-Growth algorithm is approximately one order of magnitude faster than the Apriori algorithm, but it has two disadvantages： the first is that its frequent pattern tree may be too big to be created in the memory ; the second is its serial processing approach. In this paper we propose a kind of distributed parallel association rule mining algorithm. It is for the distributed applied data framework, does not need to create the global FP-tree so avoids the problem of too big the global FP tree that fills the memory to excess. In all its principal steps the algorithm achieves parallel processing. Test resuh and analysis of the algorithm show that compared with conventional association rule mining algorithm FP-Growth, this one significantly improves the executing efficiency and the processing ability by multi-node distributed parallel processing.