融合类别特征扩展与N-gram子词过滤的fastText短文本分类

Short Text Classification with Category Feature Expansion and N-gram Subword Filtration Based on fastText

以提升fastText短文本分类模型性能为目标,从获取高质量的类别特征、降低N-gram子词中低类别区分贡献度子词对模型学习高类别区分贡献度语义特征时产生的干扰角度展开研究,提出基于TF-IDF的LDA类别特征提取方法以提升类别特征质量,提出基于词汇信息熵的N-gram子词过滤方法过滤N-gram子词中低类别区分贡献度子词,并构建更专注于高类别区分贡献度语义特征学习的EF-fastText短文本分类模型.实验结果表明基于TF-IDF的LDA类别特征提取方法,以及基于词汇信息熵的N-gram子词过滤方法对于EF-fastText短文本分类模型性能提升是有效性的.

In order to improve the performance of fastText short text classification model,the research which gets the higher quality category features,and reduces the interference of n-gram subwords with low category distinction contribution to model learning semantic features with high category distinction contribution,is carried out.A LDA category feature extraction method based on TF-IDF is proposed to improve the quality oncategory feature,a N-gram subwords filtering method based on lexical comentropy is proposed to filter the subwords with low category distinction contribution in the n-gram subwords list,and a short text classification model named by EF-fastText that focuses more on the learning of semantic features with high classification contribution is constructed.The experimental results show that the LDA extract category feature method based on TF-IDF and the N-gram subwords filtering method based on lexical comentropy are effective in improving the performance of EF-fastText short text classification model.