基于改进的Jaccard系数文档相似度计算方法

Text Similarity Method Based on the Improved Jaccard Coefficient

文本相似度主要应用于学术论文查重检测、搜索引擎去重等领域,而传统的文本相似度计算方法中的特征项提取与分词环节过于冗杂,而且元素的随机挑选也会产生权重的不确定性.为了解决传统方法的不足,提出一种基于改进的Jaccard系数确定文档相似度的方法,该算法综合考虑了各元素、样本在文档中的权重及其对多个文档相似度的贡献程度.实验结果表明,基于改进的Jaccard系数的文档相似度算法具有实效性并且能够得到较高的准确率,适用于各种长度的中英文文档,有效地解决现有技术中存在的文档间相似度计算不精的问题.

Text similarity check is mainly used in Re-check detection of Papers, the deduplication of search engines and other fields. However, it＇s extremely fussy to extract feature items with the traditional methods for computing the text similarity. In addition, it will bring uncertainty to select elements randomly. To solve these problems, a text similarity method based on improved Jaccard coefficient is proposed. This method takes into account the weights of elements and samples in the document, even the contribution degree to multiple text similarity. The results suggest that the text similarity method based on the improved Jaccard coefficient has been proved to be effective with a satisfactory accuracy, which can be applicable to various lengths of Chinese, English documents. It effectively solves the problem of inexact computing with existing technologies.