基于自注意力机制的双向分层语义模型

Bi-directional hierarchical semantic representation model based on self-attention mechanism

针对以文本词向量作为输入的神经网络无法充分利用文本语义结构特征信息、难以有效表示每个词语在句子中的重要程度等问题,提出一种基于自注意力机制的双向分层语义模型Satt-BiHSNN。通过双层双向LSTM将文本词向量进行训练得到文本表示,解决长距离依赖问题;通过自注意力机制从多个视角有效获取每个词语在句子中的重要程度,减少噪音词语权重并获取更多隐藏信息;使用softmax分类器进行文本分类。在IMDB和20Newsgroup数据集上的实验结果表明,该方法在文本分类任务中,较之前基于传统注意力机制的文本分类模型在准确率和收敛速度上有了进一步的提高。

To address the problem that the neural network with word vector as input cannot make full use of text semantic structure feature information and it is difficult to effectively represent the importance of each word in sentences,a hierarchical semantic representation model called bi-directional hierarchical semantic neural network based on self-attention(Satt-BiHSNN)was proposed.The text word vector was trained using double-layer bidirectional LSTM to obtain the text representation and solve the problem of long-distance dependence.The importance of each word in the sentence was effectively obtained through the self-attention from multiple aspects,and the weight of the noise words was reduced and more hidden information was got.The softmax classifier was used for text categorization.Experimental results on IMDB and 20Newsgroup datasets show that,compared to the traditional attention mechanism,the proposed model further improves the accuracy and convergence speed of the text classification model.