基于多SimHash指纹的近似文本检测

Efficient Near-duplicate Detection Based on Multiple SimHash Fingerprints

近似文本检测已成为当前研究热点.基于SimHash指纹的近似文本检测是主流的检测方法之一.但使用SimHash进行近似文本检测存在如下问题:指纹位数单一,丢失了一定量的信息.针对该问题,为使SimHash指纹尽可能多地代表文档的内容或特征,通过对术语集的统计特征分析,提出基于多SimHash指纹和k维超曲面的近似文本检测算法.实验表明基于多Sim-Hash指纹的近似文本检测算法提高了检测的准确率,而且所增加的时间代价很小.

Near-duplicate detection has attracted significant attention over the past years. The near-duplicate detection based on Sim-Hash is one of the state-of-the-art algorithms. However, there exists a problem for this method： SimHash maps high-dimensional vectors to small-sized and well formatted （fixed length） fingerprints, which lost a certain amount of information. To solve the problem, this paper firstly introduces the analyses of statistical characteristics of term sets. Then a novel and efficient near-duplicate detection scheme based on multiple SimHash fingerprints and k-dimensional hypersurfaees is presented. Experimental results prove that the scheme can significantly improve the precision and F1, and execution times are almost remained unchanged.