一种结合词项语义信息和TF-IDF方法的文本相似度量方法

A Text Similarity Measurement Combining Word Semantic Information with TF-IDF Method

传统的文本相似度量方法大多采用TF-IDF方法把文本建模为词频向量,利用余弦相似度量等方法计算文本之间的相似度.这些方法忽略了文本中词项的语义信息.改进的基于语义的文本相似度量方法在传统词频向量中扩充了语义相似的词项,进一步增加了文本表示向量的维度,但不能很好地反映两篇文本之间的相似程度.文中在TF-IDF模型基础上分析文本中重要词汇的语义信息,提出了一种新的文本相似度量方法.该方法首先应用自然语言处理技术对文本进行预处理,然后利用TF-IDF方法寻找文本中具有较高TF-IDF值的重要词项.借助外部词典分析词项之间的语义相似度,结合该文提出的词项相似度加权树以及文本语义相似度定义计算两篇文本之间的相似度.最后利用文本相似度在基准文本数据集合上进行聚类实验.实验结果表明文中提出的方法在基于F-度量值标准上优于TF-IDF以及另一种基于词项语义相似性的方法.

Traditional text similarity measurements use TF-IDF method to model text documents as term frequency vectors,and compute similarity between text documents by using cosine similarity.These methods ignore semantic information of text documents,and semantic information enhanced methods distinguish between text documents poorly because extended vectors with semantic similar terms aggravate the curse of dimensionality.This paper proposes a similarity measurement,which is based on TF-IDF method,and analyzes similarity between important terms in text documents.This approach uses NLP technology to pre-process text,and uses TF-IDF method to filter those key terms that have higher TF-IDF value than other common terms.With the proposed data structure TSWT（Term Similarity Weight Tree） and the definition of semantic similarity,this paper resolves the semantic information of those key terms to compute similarities between text documents.Finally,several K-Means clustering methods is used for evaluating performance of the new text document similarity.By comparing with TF-IDF and another the-state-of-art semantic information based similarity method,experimental results on benchmark corpus demonstrate that it can promote the evaluation metrics of F-Measure.