基于注意力掩码语言模型的隐式篇章关系识别

Attention-based Mask Language Model for Implicit Discourse Relation Classification

隐式篇章关系识别是在缺少显式连接词的条件下,自动判定论元之间的语义关系。其挑战性在于现有训练数据的规模较小,其蕴含的语义多样性也相对有限。针对上述问题,该文利用掩码语言模型架构建立篇章关系分类模型。其动因包括:①掩码语言模型在自监督学习过程中具备局部的语言生成能力,即在理解上下文语义的基础上“重构掩码区域语义表示”的能力;②掩码重构形成了数据增强(潜在的自动数据扩展)的效果,有助于提高篇章关系分类模型的鲁棒性。特别地,该文提出一种基于交互注意力的掩码语言模型,该方法计算论元之间的交互注意力矩阵,并依赖交互注意力动态选择论元之间高关联性的关键词项进行遮蔽、掩码重构,从而形成更有针对性的数据增强(非关键信息的数据增强对关系分类影响不大)。该文利用宾州篇章树库语料进行实验。实验结果表明,相较于基准系统,我们提出的方法的F 1值在四大类关系(对比关系、偶然性关系、扩展关系和时序关系)上分别提高了3.21%、6.46%、2.74%和6.56%。

Implicit discourse relation recognition is to determine the semantic relations between arguments in the absence of explicit connectives.The challenge lies in the small scale of the existing training data and the relatively limited semantic diversity contained in it.To address the issue,this paper proposes a novel discourse relation recognition method based on the interactive-attention-based mask language model.The motivations include①the mask language model has local language generation capabilities in the self-supervised learning process,that is,the ability to"reconstruct the semantic representation of the mask region"based on understanding the contextual semantics;②the mask reconstruction has formed the effect of data enhancement(potentially automatic data expansion)and improves the robustness of discourse relation recognition.Technically,the method calculates interactive-attention weights between the arguments.Then,we select the keywords between arguments for masking according to interactive-attention weights.The experiments on Penn Discourse Treebank 2.0(PDTB 2.0)show that the proposed method increases F1 score by 3.21%,6.46%,2.74%,and 6.56%for four top relations(Comparison,Contingency,Expansion,and Temporal),respectively.