Spark在集成学习文本情感分析中的应用

SPARK S APPLICATION IN TEXT EMOTIONAL ANALYSIS OF ENSEMBLE LEARNING

针对使用集成学习方法进行大规模文本情感分析实验中计算时间瓶颈的问题,提出基于Spark平台的集成学习模型并行化算法。使用三个数量级的文本进行集成学习的对比实验。结果表明,该算法大幅缩短了文本分类时间,F-score等相关评价指标与单机版本接近,且算法的可拓展性良好,大幅降低了模型优化和调参的时间成本。

Aiming at the problem of computing time bottleneck in large-scale text emotional analysis experiments using ensemble learning,this paper proposes a parallel algorithm of ensemble learning model based on Spark platform.Three order of magnitude texts were used to conduct a comparative experiment of ensemble learning.Experiments show that our algorithm greatly shortens the time of text categorization,and F-score and other related evaluation indicators are close to the single version.And it also has good extensibility,which greatly reduces the time cost of model optimization and parameter adjustment.