基于关键词聚类的新闻文本相似度计算

News Text Similarity Calculation based on Keyword Clustering

针对新闻文本篇幅长、冗余信息多、文本相似度难以准确高效计算的问题,提出一种基于关键词聚类的新闻文本相似度计算方法。首先对文本数据进行预处理,挖掘出文本中的关键信息。使用以TF-IDF值为权重的加权采样方法抽取文本数据集中的关键词,基于聚类的方法光滑噪声数据。聚类形成簇后,在簇间词语相似度计算上,使用word2vec融合TF-IDF词语加权的计算方法,同时关注词语间的语义信息和词语频率。最后,基于各簇的相似度计算两篇文本的相似度。实验表明,所提新闻文本相似度计算方法在计算效果上优于传统计算方法。

Aiming at the problems of long news text,too much redundant information,and difficulty in accurately and efficiently calculating text similarity,a news text similarity calculation method based on keyword clustering is proposed.First,the text data is preprocessed to extract the key information in the text.The weighted sampling method weighted by TF-IDF values was used to extract keywords in the text dataset,and the clustering-based method was used to smooth noise data.After getting clusters from clustering,word2vec is used to calculate the word similarity between clusters,and the TF-IDF word weighting calculation method is used,and the semantic information and word frequency between words are considered.Finally,the similarity of the two texts is calculated based on the similarity of each cluster.Experiments show that the proposed news text similarity calculation method performs better than the traditional calculation method.