基于粗糙集的海量数据挖掘算法研究

Research on massive data mining algorithm based on rough set

针对传统数据挖掘算法在数据量级方面的局限性,提出在粗糙集理论的基础上,采用类分布链表结构改进传统的基于属性重要性的数据离散化算法、属性约简算法以及基于启发式的值约简算法;讨论了基于动态聚类的两步离散化算法,当算法适应大数据处理之后,采用并行计算的方法提高算法的执行效率。算法测试结果表明改进的算法能有效地处理大数据量,同时并行计算解决了大数据量处理带来的效率问题。

Since traditional data mining algorithms have the limitation of data magnitude,and on the basis of rough set theory,the class distribution list structure is used to improve the traditional data discretization algorithm based on attribute im-portance,attribute reduction algorithm and heuristic-based value reduction algorithm. A two-step discrete algorithm based on dynamic clustering is discussed. When the algorithm is suited for the big data processing,the parallel computing method is used to improve the execution efficiency of the algorithm. The test results of this algorithm show that the improved algorithm can process the massive data effectively,and the parallel computing can solve the efficiency problem caused by massive data processing.