基于Low-IDF-SIG的句子重复检测

Sentence Near-Duplicate Detection Based on Low-IDF-SIG

随着互联网上数据的爆炸式增长,互联网上产生了大量的重复数据。这些重复数据给搜索引擎、观点挖掘等许多Web应用带来了严峻的问题。目前绝大部分的重复检测的算法均着重考虑文档级别,不能有效地检测出两个文档中只有一部分互为拷贝的情况。而句子级别的重复检测正是解决这类问题的一个必要步骤。该文提出了一种快速有效的句子级别的特征抽取方法——Low-IDF-Sig算法,算法依据选定的先行词从句子中抽取出改进的Shingle特征以表示句子内容。真实语料库上的实验结果证明该文提出的算法能有效地提高句子级别重复检测任务的效率和精度。

Because of the explosion of the Internet,enormous duplicated data cause serious problem for search engine,opinion mining and many other Web applications.Most existing near-duplicate detection approaches focus on the document level,incpapble of finding out the duplicated part that is just a small piece of both documents.Near-duplicate detection on sentence level is a key solution to such problem.An effective and efficient feature extraction algorithm namedLow-IDF-Sig is proposed in this paper.In order to express a specified sentence,our algorithm extracts the improved Shingle feature according to selected antecedents.Experimental results based on a real corpus show that our proposed method can improve both precision and efficiency of near-duplicate detection in sentence level.