基于向量空间模型的中文文本相似度的研究

Research on Chinese text similarity based on vector space model

在文本聚类中需要衡量中文文本之间的相似性。本文首先讨论了文本相似度的概念和常用计算算法,详细介绍了向量空间模型和算法步骤,采用删除去除词表、近义词合并、修改文件长度3中策略对算法进行了改进。最后借助盘古分词组件和搜狗实验室的互联网词库,在Visual Studio 2008环境下使用C#语言对算法进行了实现。使用在CNKI上得到的5个不同领域的500篇学术论文的中文摘要对算法进行了测试,结果表明新算法在误差率方面有较大改善,但运行时间较长。

In text clustering, the similarity between the Chinese text needs to be measured. Firstly, this paper discusses the concept of text similarity and common algorithm, vector space model and steps of the algorithm are introduced in detail, using a stoplist removal and merger of synonyms, modify file length 3 strategies to improve the algorithm. Finally With the help of Internet of Pangu word components and Sogou laboratory thesaurus, under the environment of Visual Studio 2008 using C#language algorithm is implemented. The algorithm was tested using the 500 academic papers on the CNKI obtained from 5different fields. The results show that new algorithm in the error rate is improved, but the running time is longer.