基于LDA模型的大规模文本挖掘算法研究

An Research on Large Scale Text Mining Algorithm Based on LDA Model

针对海量文本信息的挖掘与处理是目前文本挖掘领域的一个热点问题,LDA模型是文本挖掘领域中应用广泛的一种主流模型。LDA模型能够通过快速学习和训练,检测出文本集中潜在的主题词。传统算法对速度和内存优化并不明显。文章从几个不同的方面阐述了LDA模型及其扩展模型在处理大规模文本数据时的应用,从参数估计、在线学习和并行优化方面进行了比对,对其中在线模型进行了实验验证,引入狄利克雷参数、BP信念传播算法,在几组数据集上进行了实验,分析其性能。

The mining and processing of massive text information is a hot issue in the field of text mining.The LDA model is a mainstream model widely used in this field.The LDA model is able to detect potential subject terms in the text set through rapid learning and training.Traditional algorithms are not obvious for speed and memory optimization.The article expounds the application of LDA model and its extended model in dealing with large-scale text data from several different aspects.It compares parameter estimation,online learning and parallel optimization.The online model is experimentally verified and introduced.Dirichlet parameters and Belief Propagation were tested on several sets of datasets and the performance of the online LDA model was analyzed.