中文文本聚类常用停用词表对比研究

Chinese Stopwords for Text Clustering:A Comparative Study

【目的】通过实验对比分析,比较不同停用词表对于不同类型的文本数据的作用效果,对停用词表的构建与使用提供参考意见。【方法】选取百度停用词表、哈尔滨工业大学停用词表以及四川大学机器智能实验室停用词表,基于三个不同语料库运用汉语分词技术、TF-IDF特征评估函数以及VSM模型进行文本处理,并且采用Java编写的K-means算法进行聚类实验,通过准确率P、召回率R和F1三个评价指标对不同聚类结果进行效果评估。【结果】不同停用词表对于不同类型的文本数据作用效果差异明显,词表的长度、内容结构是影响作用效果的直接因素,其中两字停用词作用效果最为明显。【局限】实验文本类型及数量有限,同时对于不同停用词表仅在词语数量及内容上做了简单的分析比较,未对停用词按照类别分类进行实验分析。【结论】停用词表对于文本聚类准确度有很大的影响,构建或选取适宜的中文停用词表极为重要。同时,过度增加停用词的数量并不会一直改善聚类结果。

[Objective] This paper compares and analyzes the impacts of stopwords on textual data processing, aiming to improve the construction and use of stopwords. [Methods] We obtained stopword lists from Baidu Search Engine, Harbin Institute of Technology and the Machine Learning Laboratory of Sichuan University for this study. First, we processed text message with the stopword lists and Chinese word segmentation technique, the TF-IDF feature evaluation function and the VSM vector model. Secondly, we analysed the texts with the K-means algorithm to calculate the P, R and F1 values. [Results] Different stopword lists posed various effects to the text data processing tasks. The length of the list and the content structure of the texts directly influenced the clustering results. More importantly, the two-character stopwords was the biggest factor. [Limitations] The text types and quantity were limited. More research is needed to analyze the text with different types of stop words. [Conclusions] Stopword list poses significant impacts on text clustering, thus, it is extremely important to build or choose the appropriate Chinese stopword list. However, excessively increasing the number of stop words might not always improve the clustering results.