基于ERNIE 2.0模型的用户评论多标签文本分类研究

Research on Multi Label Text Classification of User Comments Based on ERNIE 2.0 Model

文章针对多标签文本分类这一热点问题,采用“预先训练模型+微调策略”模式,即研究持续学习语义理解框架ERNIE 2.0和基于知识蒸馏的压缩模型ERNIE Tiny预先训练模型,以及倾斜的三角学习率STLR微调策略在用户评论多标签文本数据集中的实践。相对经典语义表征模型BERT,采用ERNIE 2.0模型的效果可提高1%以上,采用ERNIE Tiny模型的速率可提升3倍左右;相对默认微调策略,采用倾斜的三角学习率STLR微调策略的效果同样可再提高1%左右。

Aiming at the hotspot issue of multi label text classification,this paper adopts the mode of“pre training model+fine tuning strategy”,that is,to study the continuous learning semantic understanding framework ERNIE 2.0,the compression model ERNIE Tiny pre training model based on knowledge distillation,and the practice of inclined triangular learning rate STLR fine tuning strategy in user comments multi label text data sets.Compared with the classical semantic representation model BERT,the effect of ERNIE 2.0 model can be improved by more than 1%,and the rate of ERNIE Tiny model can be increased by about 3 times;compared with the default fine tuning strategy,the effect of inclined triangular learning rate STLR fine tuning strategy can also be improved by about 1%.