基于主题和特征的文本相似度算法研究

The Research of Texts' Similarity Algorithm Based on Topic and Features

本文提出了结合主题和各主题下关键特征的文本相似度算法,目的在于更准确的挖掘被描述对象的近邻对象集。本文首先介绍卡方统检验特征统计法,并利用改进的卡方检验,计算训练集中已知主题的文本的特征;而后介绍了最小编辑距离算法、余弦相似度算法和杰卡德相似系数,在论证了主题对文本相似度的重要性后,又针对难提取主题的文本加以改进,最终提出了基于主题和特征的文本相似度算法;然后对各个算法在测试集上的相似度计算结果进行分析,证明本文提出的算法在速度和精确度上明显优于其他算法;最后将该算法应用于股票的概念股题材标注上,分析结果并提出改进空间和不足之处。

In this paper, we proposes a texts' similarity algorithm based on the topic and features of each text, the purpose is to accurately mine the nearest neighbor texts of a given text. Firstly, we introduces the characteristics of CHI Square Statistics and gives improvement to features selection of training text which have known topic; and then compare the minimum edit distance algorithm(Leven Shtein Distance), Cosine Similarity Algorithm and Jaccard Similarity Coefficient, analyze principal defect of each algorithm and propose a new text similarity algorithm based on features and topic; after that we use the new algorithm on real data set and prove no matter in speed or accuracy the new one is better than others; In the end, the new algorithm is applied to stocks' subject tagging, from the analysis of results, we expound recent shortage and put forward the improvement.