基于Word2vec的微博短文本分类研究

Research of Weibo Short Text Classification Based on Word2vec

随着微博等社会化媒体的信息量急剧膨胀,人们迫切需要实现这些信息的自动分类处理,以帮助用户快速查找所需信息和过滤垃圾信息。针对传统文本分类模型存在的特征维数灾难、无语义特征等问题,文章基于Word2vec模型对微博短文本进行了分类研究。鉴于Word2vec模型无法区分文本中词汇的重要程度,进一步引入TFIDF对Word2vec词向量进行加权,实现加权的Word2vec分类模型。最后合并加权Word2vec和TFIDF两种模型,实验结果表明合并后模型分类准确率高于加权Word2vec模型和使用TFIDF的传统文本分类模型。

With the rapid expansion of new available information on Microblogging and other social media. Text automatic classification becomes imperative in order to help people locate the information he inquires and filter spam. Based on the characteristics of curse of dimensionality and lack of semantic features in Traditional text classification model, put forward a short text classify based on Word2 vec model.Since Word2 vec can not distinguish the weight of words, we applied weights using tfidf weighting with Word2 vec, implemented weighted Word2 vec. Then we concatenated tf-idf with our word2 vec weighted by tf-idf. Our results show that the combination of Word2 vec weighted by tf-idf without stop words and tf-idf without stop words can outperform either Word2 vec weighted by tf-idf without stop words and tf-idf with or without stop word.