大数据质量管理:问题与研究进展

Big Data Quality Management: Problems and Progress

当前大数据在多个领域广泛存在,大数据的质量对其有效应用起着至关重要的作用,因而需要对大数据进行质量管理。尽管数据质量管理方面已经有一些研究成果,但由于大数据具有规模大、速度快和多样性高的特点,现有的方法难以适用于大数据质量管理。本文针对错误发现、错误修复和劣质数据查询处理,综述了大数据质量管理的问题与挑战,认为大数据质量管理的挑战主要有计算困难、错误混杂和缺少知识3个方面。本文依据这3个方面的解决方法,对大数据质量管理目前的研究进展进行了综述,并展望了大数据质量管理未来的研究方向。

Big data have wide applications. Since the quality of big data plays a crucial role in these data-centric applications, data quality management techniques for big data are in demand. Although some theories and techniques for data quality management have been proposed, due to the volume, variety and velocity of big data, current methods could hardly be applied to data management for big data. This paper discusses the problems and challenges for error detection, error repair and query processing of dirty data in big data management, and identifies intractability, mixed errors and the lack of knowledge as three new challenges to data quality management. The progress of big data quality management in these three aspects is reviewed and open problems for future research are proposed.