基于张量分解的分布式主题分类模型

Improved Distributed Topic Classification Model Based on Tensor Decomposition

针对大规模数据分类时计算时间长以及分类精度下降等问题,提出使用张量分解求解LDA主题模型参数,实现对海量网络数据的采集、分类、挖掘.该方法使用矩量法将LDA模型求解转化为低维的张量分解问题,通过分解和反射进行参数的传递,运用大数据平台Spark的进行分布式计算.实验结果表明,改进的模型参数计算方法在时间效率和困惑度方面都得到了提升,并且分类信息更加直观,更加适用于大规模网络数据分类工作.

Aiming at the problems of large computation time and low classification time, this study presents an improved parameter estimation model for LDA by using the method of tensor decomposition, which can collect, classify, and mine massive network data. Using the method of moments, the LDA model calculation is transformed into low-dimensional tensor decomposition, and the parameters are transferred by decomposition and reflection. The large data platform Spark is used for distributed computation. The experimental results show that the model has been improved in terms of running time and perplexity, and the classification information display is more intuitive, which is more suitable for large-scale network data classification.