基于LDA模型的中文微博热点话题发现

A Hot Topic Identification based on LDA for Chinese Microblog

针对微博文本数量增加速度快、信息量繁杂等问题,将LDA模型应用到热点话题的挖掘中,构建出微博热点话题的识别过程。首先应用LDA模型对微博语料库进行主题建模,采用困惑度方法确定最佳主题个数,通过Gibbs抽样算法实现参数推理,获得语料库的主题-词汇概率分布和文本-主题概率分布,在此基础上计算并识别出微博中的热点话题、热点词汇和热点话题微博。实验结果显示该模型与人工挑选的结果基本一致,表明该模型具有较好的热点识别效果。

In order to solve the problem that the number of microblog text is increasing quickly and the amount of microblog information is very complicated, LDA model is applied to mine the hot topic, and the identification process of mieroblog hot topic is constructed. Firstly, we use LDA to model microblog corpus, determine the best number of topics by the perplexity, and achieve parameters estimation with Gibbs sampling algorithm, then we obtain the probability distribution of the topic and the word and the probability distribution of the text and the topic, on the basis of this, we calculate and identify hot topics, hot words and hot topics microblog. Experimental results show that this model is consistent with the results of artificial selection, indicating that the model has better recognition performance on hotspots.