一种结合TF-IDF和Simhash的科技项目文本相似性度量方法

An approach for text similarity measurement of science and technology projects combing TF-IDF and Simhash

为了提高科技项目文本相似性度量的准确性和性能,将TF-IDF和Simhash相结合,提出了一种新的科技项目文本相似性度量方法。首先,该方法对科技项目文本进行预处理得到词项集合,再使用TF-IDF计算词项集合中每个词项的权重值,并选取具有较高权重值的重要词项;其次,使用Simhash把重要词项映射为固定长度的二进制串,并求和得到文本的Simhash签名;最后,使用汉明距离计算两个Simhash签名间的相似性。实验结果表明,所提方法在查准率、召回率和F度量值方面优于传统的Simhash算法和TF-IDF方法。

To enhance the accuracy and performance of text similarity measurement of science and technology projects,this pa‐per proposes a new approach for measuring text similarity of science and technology projects by combining TF-IDF and Simhash.Firstly,this method uses natural language processing technology to preprocess science and technology project texts to get a term set,then uses the TF-IDF method to calculate the TF-IDF value of each term in the term set,and selects the important term with higher TF-IDF value.Secondly,this method uses the Simhash algorithm to get the Simhash signature of the text through mapping the selected important terms into fixed binary strings.Finally,Hamming distance is used to calculate the similarity between two Simhash signatures.Experimental results show that compared to the traditional Simhash and TF-IDF,the proposed method can promote the evaluation metrics of precision,recall and F-measure.