基于网格分组和属性权值的相似重复记录识别算法

An Improved Approximately Duplicate Records Detection Algorithm

针对在处理海量数据时,传统的相似重复记录识别方法具有检测效率不高、检测精度较低等缺陷,提出了一种基于网格分组和属性权值的相似重复记录检测算法。该算法采用分而治之的思想,使用网格法将海量数据进行分组,并为各属性设立相应的权值,然后进行相似重复记录的识别。理论分析和实验表明,网格分组有效减少了记录之间的比对次数,基于属性综合权值的方法更加准确反映各属性对记录的贡献度,两者结合能够有效解决大数据的相似重复记录的识别问题。

The traditional detection algorithm has bad efficiency and low detecting precision on approxi-mately duplicate records when it deals with massive data. An improved algorithm based on the grid-based grouping and at tribute weights is proposed. The divide-and-conquer theory is used in this algorithm, and the grid method is used to group the massive data, and it sets up corresponding weights for all the attrib-utes to identify the duplicate records. Both theory and experimental show that the grid group can reduce the comparisons effectively between the records, and the method based on at tribute synthetic weights re-flects the contribution of each at tribute to the record more accurately, and a combination of both can solve the problems of recognizing the big data？s duplicate records effectively.