面向问题生成的预训练模型适应性优化方法研究

Adaptive Optimization Method of Pre-trained Language Model for Question Generation

问题生成的核心任务是“在给定上下文语境的前提下,对目标答案自动生成相应的疑问句”。问题生成是自然语言处理领域中富有挑战性的任务之一,其对可靠的语义编码和解码技术有着极高的要求。目前,预训练语言模型已在不同自然语言处理任务中得到广泛应用,并取得了较好的应用效果。该文继承这一趋势,尝试将预训练语言模型UNILM应用于现有“基于编码和解码架构”的问题生成系统中,并集中在其适应性问题上开展研究。该文针对预训练模型在解码阶段频繁出现的“暴露偏差”和“掩码异构”问题,分别研究了基于随机抗噪和迁移学习的训练方法,借以提升UNILM在问题生成过程中的适应能力。同时,该文在SQuAD数据集上开展问题生成实验,实验结果证明,随机抗噪和迁移学习都能优化UNILM的解码性能,使之在答案可知场景的数据划分split1和split2上,分别将BLEU4指标提升到20.31%和21.95%;并在答案不可知场景的split1数据集上将BLEU4指标提升到17.90%。

Automatically question generation(QG for short)is to automatically generate the corresponding interrogative sentence of the target answer under the given context..In this paper,we take advantage of pre-trained language model and apply the UNILM on encoder-decoder framework of question generation.In particular,in order to solve the problems of"exposure bias"and"mask heterogeneity"in the decoding phase of model,we examine the noise-aware training method and transfer learning on UNILM to raise its adaptability Experiments on SQuAD show that our best model yields state-of-the-art performance in answer-aware QG task with up to 20.31%and 21.95%BLEU score for split1 and split2,respectively,and in answer-agnostic QG task with 17.90%BLEU score for split1.