基于开放域对话系统的自动化评测方法研究

Automatic evaluation method for open domain dialogue systems

开放域对话系统的研究在近年来取得了很大的进展,然而基于该类系统的自动化评测依然是目前亟待解决的问题。针对目前各类评测方法需要大量标注数据和评测准确率较低等问题,提出了一种利用长短期记忆网络和注意力机制判别问题-回复对是否为真实对话的评测模型。该模型基于连续的对话语料进行建模,解决了目前基于参考回复的评测模型需要大量标注数据的弊端。在Cornell和Reddit数据集上,该模型分别取得了57.2%和71.8%的准确率,与现有几种评测模型相比准确率有明显提升。

Although great progress has been made in open domain dialogue systems in recent years,automatic evaluation methods based on these systems are still a problem to be solved.In order to solve the problem that various evaluation methods need a lot of tagged data and low accuracy,this paper proposed a model for judging whether the response pair was a real dialogue by using the long-term and short-term memory network and attention mechanism.The model was based on continuous dialogue corpus,which solved the shortcomings of the current evaluation methods based on the reference response.On the Cornell and Reddit data sets,the accuracy of the model is 57.2%and 71.8%respectively,which is obviously improved compared with the existing evaluation models.