基于分布式计算模式的两种属性约简算法

Two Attribute Reduction Algorithms with Distributed Computing Pattern

粗糙集是一种处理不确定、不完全理论的经典理论,属性约简是粗糙集理论的核心知识之一。为适应大数据的发展,对广泛应用于数据不确定性、不完备性处理的属性约简算法相应改进,提出两种适应分布式处理的属性约简算法。首先提出基于MapReduce框架和正域的属性约简算法;并借鉴Hadoop分布式处理机制,针对差别矩阵all-to-all比较特性,提出一种新型分布式处理机制,基于该机制提出了一种适应分布式数据处理机制的新型数据分割及分布规则,基于该规则探讨了基于差别矩阵的属性约简算法在新型分布式处理机制下的具体实现方法。仿真算例表明了这两种算法处理大数据集的有效性。

Rough set is a classical theory of processing uncertain and incomplete theory, and attribute reduction is one of the core knowledge of rough set theory. In order to adapt to the development of big data, the attribute reduction algorithm should be improved accordingly with abroad application in data uncertainty and incomplete processing. In this context, we propose two kinds of attribute reduction algorithm. First, an attribute reduction algorithm based on the MapReduce framework and the positive domain is presented. Then, a new distributed processing mechanism is proposed for the all-to-all comparison feature of the difference matrix by means of Hadoop distributed processing mechanism, based on which, a new data segmentation and distribution rules of the data processing mechanism are discussed. On the basis of this rule, the attribute reduction algorithm based on the difference matrix is discussed in the new distributed processing mechanism. A simulation example shows the effectiveness of the two algorithms in dealing with large data sets.