多模态命名实体识别方法研究进展

Research Progress of Multimodal Named Entity Recognition

为了解决多模态命名实体识别(MNER)研究中存在的文本特征语义不足、视觉特征语义缺失、图文特征融合困难等问题,多模态命名实体识别方法相继被提出。首先,总结了多模态命名实体识别方法的整体框架以及各部分常用的技术,随后对其进行梳理并分类为基于BiLSTM的MNER方法和基于Transformer的MNER方法,并根据模型结构将其划分为前融合模型、后融合模型、Transformer单任务模型、Transformer多任务模型等4类模型结构。其次,在Twitter-2015、Twitter-2017 2个数据集上,分别对这2类方法进行实验,结果表明:多特征协同表示能增强各模态特征的语义,多任务学习能够促进模态特征融合或者结果融合,从而提升MNER的准确性。建议在MNER的未来研究中,着重关注通过多特征协同表示来增强模态语义,通过多任务学习促进模态特征融合或结果融合等方向的研究。

In order to solve the problems in studies of multimodal named entity recognition,such as the lack of text feature semantics,the lack of visual feature semantics,and the difficulty of graphic feature fusion,a series of multimodal named entity recognition methods were proposed.Firstly,the overall framework of multi modal named entity recognition methods and common technologies in each part were examined,and classified into BilSTM-based MNER method and Transformer based MNER method.Furthermore,according to the model structure,it was further divided into four model structures,including pre-fusion model,post-fusion model,Transformer single-task model and Transformer multi-task model.Then,experiments were carried out on two data sets of Twitter-2015 and Twitter-2017 for these two types of methods respectively.The experimental results showed that multi-feature cooperative representation could enhance the semantics of each modal feature.In addition,multi-task learning could promote modal feature fusion or result fusion,so as to improve the accuracy of MNER.Finally,in the future research of MNER,it was suggested to focus on enhancing modal semantics through multi-feature cooperative representation,and promoting model feature fusion or result fusion by multi-task learning.