基于LDA改进的K-means算法在短文本聚类中的研究

Improved K-means algorithm based on Latent Dirichlet Allocation for short text clustering

在短文本聚类的过程中,常发现特征词的稀疏性质、高维空间处理的复杂性.由于微博的内容长度限制和特征稀疏性,特征向量的高维度被执行,导致模糊聚类结果.本文使用了Latent Dirichlet Allocation主题模型,对训练数据进行建模,并将主题术语扩展原始微博的特征,从而丰富了聚类文本特征,提高聚类效果.实验结合K-means和Canopy聚类算法对文本数据进行处理,提出了LKC算法,弥补了K-means算法对初始聚类中心点选取的敏感性,结果实现了更高的精度和聚类F1-measure的测量值.F1值提高了10%,准确度提高了2%.

In the process of short text clustering, the sparse nature of the characteristic words, the complexity of the high- dimensional space processing are often found. Due to the content length limitation of the micro blog and its feature sparsity, the high dimensionality of feature vectors is performed, resulted in obscured clustering results. A Latent Dirichlet Allocation （LDA）theme model is proposed to the training data, and extend the subject term into the characteristics of the original micro blog, such that to enrich the category features to improve the clustering consequent. Our experiment combines K-means and Canopy clustering algorithm to process the text data and the results achieve higher accuracy and Fl-measure.The F1 value improved by 10%, and the accuracy improved by 2%.