融合BERT和AM的胶囊网络多标签文本分类

Research on Multi-label Text Classification Model of Capsule Network Based on BERT and AM

针对目前胶囊网络对特征的提取忽视特征信息在文档中的位置和远距离依赖的问题,提出1种新型融合来自变换器的双向编码器表征量(bidirectional encoder representations from transformers,BERT)和注意力机制(attention mechanism,AM)的胶囊网络模型(BA-CapsNet)。首先,通过BERT预训练模型获得全局特征词向量;然后使用多头注意力机制,对重要单词进行权重优化;最后通过胶囊网络提取局部特征,形成特征向量。同时,对动态路由算法进行改进,较好地减少低层胶囊与高层胶囊之间的信息冗余。结果表明,相比传统的胶囊网络、序列生成模型(sequence generation model,SGM)和结合卷积神经网络(convolutional neural network,CNN)的大规模多元标签文本分类(extreme multi-label text classification,XML-CNN)模型,提出的改进模型在多标签文本分类中准确率有所提升;并且相比原胶囊网络,改进的动态路由算法在模型效率上提升了约37%。

Aiming at the problem that the current capsule network ignores the position and long-distance dependence of feature information in the document for feature extraction,this paper proposes a new capsule network model(BA-CapsNet)that combines bidirectional encoder representations from transformers(BERT)and attention mechanism(AM).Firstly,the global feature word vector is obtained by BERT pre-training model.Then the multi-head attention mechanism is used to optimize the weight of important words;finally,local features are extracted by capsule network to form feature vectors.At the same time,this paper improves the dynamic routing algorithm to reduce the redundancy of information between low-level capsules and high-level capsules.The experimental results show that the improved model proposed in this paper improved the accuracy of multi-label text classification compared with the traditional capsule network,sequence generation model(SGM)and extreme multi-label text classification-convolutional neural network(XML-CNN)model,and the improved dynamic routing algorithm also improves the training efficiency of the model by about 37%compared with the original capsule network.