结构化Web数据的自动去重方法

AUTOMATIC DUPLICATION DELETION METHOD FOR STRUCTURED WEB DATA

针对载有结构化数据的网页特点,提出了一种新的有效字段发现策略,据此设计了一个基于学习的自动去重方法。对样本网页集进行聚类分析并生成每类网页的包装器,识别出包装器中的有效数据字段;对有效数据字段进行映射,通过计算有效数据字段内容的相似度来判断网页是否重复。实验证明该方法对结构化Web数据的去重有很好的召回率和准确率。

In this paper we present a new strategy of discovering valid data fields in light of the characteristic of webpage with structured data,and design a learning-based automatic duplication deletion method according to it.Sample webpage set is clustered and analysed and the wrappers of each kind of webpages are generated,and valid data fields in the wrappers are identified and then mapped.Whether the webpages has duplicate or not is determined by calculating the similarity of valid data fields’ content.Experiments indicate that this deletion approach for duplicate structural web data has a good recall rate and accuracy.