基于Spark的大数据三枝决策分类方法

Processing Big Data with Three Way Decision Based on Spark

针对大规模的数据,借助Spark平台的分布式快速处理能力,提出了基于Spark的大数据三枝决策分类方法。该方法基于三枝决策理论,使用Spark对数据进行并行化处理。由经验数据获得数据的决策边界后,通过并行的方式进行正例和反例的判断,从而提高了在大数据集上的决策效率。采用多轮的分步决策方法提高了决策的效率与准确率。通过在UCI公开数据集mushroom和connect-4上的试验结果表明,新方法适用于大数据情况下的决策问题,大大提高了三枝决策分类算法的效率。

Aiming to solve decision-making problems on big data, we combine the ability of dis-tributed data processing of spark with three-way decision theory o This method is based on three-way decision theory. The boundary of the decision regions is firstly calculated by given data, and then each sample is estimated in terms of the belongings by paralleling, which can increase the effi-ciency greatly. Multi-round step by step decision making is used to further increase the efficiency and accuracy. The experiments are conducted on the UCI datasets （mushroom and connect-4）. The results show that the proposed method is effective in processing big data.