基于注意力机制的特征融合序列标注模型

Attention based sequence labeling model with feature fusion

序列标注任务是自然语言处理领域的重要问题,包括命名实体识别、词性标注、意见抽取等具有重要应用价值的子任务。目前,长短期记忆网络-条件随机场模型(LSTM-CRF)作为序列标注任务的主流框架,虽然取得了很好的性能并被广泛使用,但仍存在局部依赖性以及受限于序列化特征学习的缺点。为了同步建模句子中每个词的局部上下文语义与全局语义,并将两部分语义进行有效融合,提出基于注意力机制的特征融合序列标注模型。具体地,本模型利用多头注意力机制建模句子中任意两个词之间的语义关系,得到每个词应关注的全局语义。考虑到LSTM学习的局部上下文信息和注意力机制学习的全局语义具有互补性,进一步设计了三种特征融合方法将两部分语义深度融合以得到更丰富的语义依赖信息。为验证模型的有效性,在四个数据集上进行了大量的实验,实验结果表明本模型达到较优的性能。

Sequence labeling is a fundamental problem in natural language processing and play a significant role in applications such as named entity recognition,text chunking,opinion extraction and so on.Even though LSTM-CRF is a widely used mainstream framework for sequence labeling in the recent years,it still has several shortcomings including local dependence and limitation of sequential learning.To synchronously model the local context semantics and global semantics of each word,and to effectively merge the two parts of semantics,an attention based sequence labeling model with feature fusion is proposed in this paper.The multi-head attention mechanism to model the semantic relation between any two words is introduced and global semantics is obtained.Considering that the context information of LSTM and the global semantics of attention mechanism are complementary,three feature fusion methods were designed to significantly fuse the two parts of semantics.Extensive experiments conducted demonstrate that our approach achieves state-of-the-art performance on four datasets.