基于Map-Reduce的大数据缺失值填充算法

Missing Value Imputation in Big Data Based-on Map-Reduce

缺失值大量存在于现实数据库中,这不仅严重影响了信息查询质量,还会扭曲数据挖掘与数据分析结论,进而误导决策.解决这一问题的最佳方法是预先填充这些丢失的数据.给出了一种基于概率推理的填充分类属性的算法.推理过程是在一个基于属性相关性而建立起来的贝叶斯网中完成.为实现大数据处理的并行化,在Map-Reduce框架中给出这两个算法.实验部分分别验证了贝叶斯网构建方法和概率推理对分类数据处理的有效性,以及算法在hadoop中运行的并行化程度.

Missing values are broadly existed in the real datasets.It affects the query quality and distorts the result of data analysis as well,which will mislead the final decision.The best way to handle this problem is to replace the missing values beforehand.This paper proposes an imputation methods based on probabilistic reasoning to deal with categorical attributes.The reasoning process are proceeded in a Bayesian network constructed according to the correlation between attributes.We complete these two algorithms using Map-Reduce for the parallelization of large data. The experimental part prove the effectiveness of the construction of Bayesian network and the probabilistic reasoning.It also gives the parallelization degree of our algorithm in hadoop.