基于模糊综合评判和长度过滤的SNM改进算法

Improved SNM Algorithm Based on Fuzzy Comprehensive Evaluation and Length Filtering

为了提高数据库的数据质量,需要对相似重复记录进行清洗,基本邻近排序算法是目前常用的清洗算法之一.针对判重过程中属性权值计算主观性过强的问题,提出通过多用户综合评判确定属性权值的方法,该方法能更客观地评判属性的重要性程度.在此基础上,结合属性权值计算两条记录的长度比例,排除不可能构成相似重复的记录,减少了比较次数,提高了检测效率.实验结果表明改进算法在查全率、查准率及时间效率等方面均有所提高.

To improve the quality of data, the approximately duplicated records need to be cleaned. The basic sorted-neighborhood method （SNM） is one of the commonly used cleaning algorithms. Aimed at the problem of excessive subjectivity of attribute weight calculation in detection algorithm, the article proposes a method based on the fuzzy comprehensive evaluation of multiuser to determine the attribute weight, which can be more objective to judge the importance level of the attribute. The proposed algorithm calculates the length ratio of the two records with attribute weight, then uses the length ratio to exclude records that are impossible to be approximately duplicated, reduces comparison times, and improves the detection efficiency. The experiment results show that the recall, precision and time efficiency are enhanced.