基于BERT的任务导向对话系统自然语言理解的改进模型与调优方法

BERT Based Improved Model and Tuning Techniques for Natural Language Understanding in Task-oriented Dialog System

任务导向对话系统的自然语言理解,其目的就是解析用户以自然语言形式输入的语句,并提取出可以被计算机所理解的结构化信息,其包含意图识别和槽填充两个子任务。BERT是近期提出来的一种自然语言处理预训练模型,已有研究者提出基于BERT的任务导向对话系统自然语言理解模型。在此基础上,该文提出一种改进的自然语言理解模型,其编码器使用BERT,而解码器基于LSTM与注意力机制构建。同时,该文提出了该模型的两种调优方法:锁定模型参数的训练方法、使用区分大小写的预训练模型版本。在基线模型与改进模型上,这些调优方法均能够显著改进模型的性能。实验结果显示,利用改进后的模型与调优方法,可以分别在ATIS和Snips两个数据集上得到0.8833和0.9251的句子级准确率。

The purpose of natural language understanding in task-oriented dialog system is to parse sentences entered by the user in natural language,extracting structured information for subsequent processing.This paper proposes an improved natural language understanding model,using BERT as encoder,while the decoder is built with LSTM and attention mechanism.Furthermore,this paper proposes two tuning techniques on this model:training method with fixed model parameters,and using case-sensitive version of pretrained model.Experiments show that the improved model and tuning techniques can bring 0.8833 and 0.9251 sentence level accuracy on ATIS and Snips datasets,respectively.