基于Spark的海量文本评论情感分析

Implementation of sentiment analysis in comments based on Spark

为实现对批量评论的情感分析,高效挖掘评论价值,该文设计并实现了基于Spark的文本评论情感分析工具。首先,该文对数据进行预处理,使各类训练数据保持平衡,并使用结巴分词对文本进行切分;其次,利用Wor2Vec模型对分词后的评论进行词向量转换;最后,将转化得到的句子向量作为分类器的输入,训练分类模型。同时,该文还基于C/S架构设计了相关应用程序,实现了批量数据的提交以及结果的快速获取。在利用Wor2Vec模型对文本进行特征提取的情况下,该文比较了几种常见分类器的性能差异,结果表明多层感知器在几种算法的比较中取得了较好的结果,能对文本情感作出较为准确的分类。

In order to realize the sentiment analysis of batch comments and improve the efficiency to dig the value of comments,we proposed a textual semantic analysis tool based on Spark platform.Firstly,we preprocessed the data and made word segmentation of comments by jieba segmentation to maintain the balance of the different training data.Secondly,with the Word2Vec model,we transformed the segmented comments into vectors that were used as input to train classifiers.We also designed related applications based on C/S architecture to submit data and acquire results quickly.Finally,we analyzed the performance of different classifiers on Spark and found that the multilayer perceptron performs best among the three classification algorithms and can judge the polarity of emotions.