结构化数据清洗技术综述

Survey of structured data cleaning methods

数据清洗是对脏数据进行检测和纠正的过程,是进行数据分析和管理的基础。该文对经典和新兴的数据清洗技术进行分类和总结,为进一步的研究工作提供方向。形式化定义了数据清洗问题,对数据缺失、数据冗余、数据冲突和数据错误这4种数据噪声的检测技术进行详细阐述。按照数据清洗方式对数据噪声的消除技术进行分类概述,包括基于完整性约束的数据清洗算法、基于规则的数据清洗算法、基于统计的数据清洗算法和人机结合的数据清洗算法。介绍了常用的测评数据集和噪声注入工具,并对未来重点的研究方向进行了探讨和展望。

Data cleaning is the process of detecting and repairing dirty data which is often needed in data analysis and management.This paper classifies and summarizes the traditional and advanced data cleaning techniques and identifies potential directions for further work.This study first formally defines the cleaning problem for structured data and then describes error detection methods for missing data,redundant data,conflicting data and erroneous data. The data cleaning methods are then summarized based on their error elimination method,including constraint-based data cleaning, rule-based data cleaning,statistical data cleaning and human-in-the-loop data cleaning.Some important datasets and noise injection tools are introduced as well.Open research problems and future research directions are also discussed.