基于并行抽样的海量数据关联挖掘算法

Mass data association mining based on parallel random sampling

在"信息爆炸"的当今社会,海量数据对数据挖掘提出新的挑战。文章针对海量数据挖掘时所面临的内存和性能问题,提出了一种基于并行随机数据抽样的云频繁项集挖掘算法。该算法可以实现在单次扫描海量数据进行并行数据抽样的基础上,对样本数据进行并行的频繁项集挖掘。实验结果表明,通过并行随机抽样算法可以有效抽取反映数据真实情况的样本数据,并对其进行相关清理,在得到样本数据后采用文中所提的并行关联云挖掘算法能有效解决内存和性能方面的问题,为推动数据挖掘在海量数据下的发展奠定了良好基础。

In an age of information explosion, data mining faces new challenges because of mass data. In view of the memory and performance problems faced by mass data mining, a cloud frequent item sets mining algorithm based on parallel random sampling of data is put forward. After the parallel da ta sampling of the single scanning mass data, it can realize the parallel frequent itemsets mining of the sample data. The experimental results prove that the parallel random sampling algorithm can effec tively extract data which can reflect the real data sets, and carry on the related cleaning. After obtai ning the sample data, the memory and performance problems can be effectively solved by this parrallel association cloud mining algorithm. This data mining algorithm lays a good foundation for promoting the development of mass data mining.