一种在线数据清洗方法

An Online Data Cleaning Method

提出一种新的在线数据清洗方法:将确认为干净的参照表中的记录字符串映射成高维空间中的点后进行聚类划分,然后利用B+树对划分中的点进行索引从而将高维空间的查询转换成一维空间的范围查询.输入表中的元组利用索引采用分枝限界策略搜索KNN(Knearestneighbors)记录从而完成与其最匹配记录的识别.理论分析和实验表明这是一种解决在线数据清洗的有效途径.

A new method for online data cleaning is presented. First, each clean record in the reference table is mapped as a point in a high-dimensional metric space measured by Manhattan distance. Next, all the points in the space are partitioned by clustering and indexed with (B+) tree. In this way, the search in high-dimensional space can be translated into search in one-dimensional space. To find the KNN (K nearest neighbors) in reference table for each incoming record, the search method of branch and bound is employed. The top K records that best match the incoming record are then identified. Theory and experiment show that it is an effective approach for online data cleaning.