基于改进的混合自注意力机制模型的研究

Research on Improved Hybrid Self-Attention Mechanism Model

自注意力机制被广泛应用于自然语言处理任务中。现有的自注意力机制通常使用软注意力模式,即使用密集注意力矩阵,但随着信息长度增长对应的权重变小,无法提取负面信息。针对以上问题,提出融合硬注意力机制即稀疏注意力矩阵来改进现有模型,充分提取文本中正负信息,弥补传统注意力矩阵不能为负的空白。

Self-attention mechanism is widely used in natural language processing tasks.The existing self-attention mechanism is soft attention mode,using intensive attention matrix.However,with the increase of information length,the corresponding weight becomes smaller and negative information cannot be extracted.Therefore,this paper use hard attention mechanism to improve the existing self attention model and fully extract the positive and negative information of text.This model not only enhances the extraction of positive information,but also makes up for the blank the traditional attention matrix cannot be negative.