摘要
目前文本去重算法主要以基于信息指纹去重、特征码去重为主,用上述算法去重时容易产生误判。本文提出了一种改进的算法,核心思想是对字符重复串进行抽取,使用重复串作为短语标引生成后缀树,并映射生成倒排索引进行STC算法去重。实验证实了改进算法有着良好的准确率和召回率,并有着优良的时间和空间特性。
Current de-duplication algorithms mainly focus on keywords de-duplication or semantic fingerprint de-duplication and may cause error when processing Web pages.This paper using the repeats as mapped sentences to make the suffix tree. Using the inverted index method to storage the data. Experiment results show that this method can find similar Web pages efficiently,this algorithm can reach a high precision in mono-language deletion of duplicated web pages, and this algorithm can also reach a maximum precision when it is applied to deletion of duplicated web pages.
出处
《微计算机信息》
2009年第27期206-208,共3页
Control & Automation
关键词
网页去重
STC算法
重复串
detect duplication technology
STC algorithm
repeats
