一种改进的短文本层次聚类算法

Improved Short Text Hierarchical Clustering Algorithm

互联网上存在着海量蕴含丰富信息的短文本数据,由于短文本存在特征稀疏、用语不规范的特点,使用传统的聚类算法效果较差。提出了一种使用词向量表示特征并结合关键词提取的短文本聚类算法:定义特征权重计算公式,计算类簇中特征的权重,得到类簇的关键词;使用Skip-gram模型训练得到的词向量计算关键词之间的语义相似度进而得到类簇的相似度实现聚类。在4个数据集上进行的实验结果表明文章的方法效果优于传统的聚类算法,宏平均较次优结果分别提高了22.3%、24.9%、2.9%和34.4%。

A large amount of short text data which contains a variety of information exits on the Internet. However,traditional clustering algorithms work poorly because of the presence of the scarcity and informality of short text. This paper presents a novel short text clustering algorithm based on feature representation with word vectors and keyword extraction. The study defines a feature weight calculation formula and calculates the feather weight of every word in clusters to get the key words to represent them. It then calculates semantic similarity between keywords based on word vectorstrained by Skip-gram model and get the similarity between clusters to cluster. Experimental results on four datasets show that this method works better than traditional clustering algorithms,with the macro-F increasing 22. 3% 、24. 9% 、2. 9% and 34. 4% compared with the second-best result.