一种基于融合递归机制的长文本机器阅读理解算法

A long text machine reading comprehension algorithm based on fusion recurrent mechanism

目前对于机器阅读理解的研究大多都使用预先训练的语言模型如BERT来编码文档和问题的联合上下文信息,相较于传统的RNN结构,BERT模型在机器阅读理解领域取得了显著的性能改进.但是当前基于BERT的机器阅读理解模型由于输入长度有限(最大长度为512),在进行特征提取时,存在一定程度的语义丢失,且不具备建立长距离依赖能力.为了解决这个问题,提出了一种基于BERT-Base的长本文机器阅读理解模型BERT-FRM.通过添加重叠窗口层以更灵活的方式切割输入文本,使用两个BERT模型独立编码问题和文档,并且在模型中添加递归层来传递不同片段之间的信息,赋予模型建立更长期依赖的能力.实验结果表明,BERT-FRM模型与BERT-Base基线模型相比,在TriviaQA和CoQA两个机器阅读理解数据集上的F1值分别提升了3.1%和0.8%.

Most of the current research on machine reading comprehension uses pre-trained language models such as BERT to encode the joint context information of documents and questions.Compared with the traditional RNN structure,the BERT model has achieved significant performance improvements in the field of machine reading comprehension.However,the current BERT-based machine reading comprehension model has limited input length(a maximum length of 512),there is a certain degree of semantic loss during feature extraction,and it does not have the ability to establish long-distance dependencies.To solve this problem,a long text machine reading understanding model BERT-FRM based on BERT-Base was presented.Overlapping window layers were included to cut input text in a more flexible way,two BERT models were used to encode problems and documents independently,and recursive layers were added in the model to convey information between different fragments,giving the model the ability to build longer-term dependencies.The experimental results showed that the F1 value of the BERT-FRM model on the TriviaQA and CoQA machine reading comprehension datasets was improved by 3.1%and 0.8%,respectively,compared with the BERT-Base baseline model.