预训练神经机器翻译研究进展分析

Pre-trained Neural Machine Translation:Progress and Analysis

神经机器翻译(NMT)模型通常使用双语数据进行监督训练,而构建大规模双语数据集是一个巨大挑战。相比之下,大部分语言的单语数据集较为容易获取。近年来,预训练模型(PTM)能够在海量的单语数据上进行训练,从而得到通用表示知识,来帮助下游任务取得显著的性能提升。目前基于预训练的神经机器翻译(PTNMT)在受限资源数据集上已被广泛验证,但如何高效地在高资源NMT模型中利用PTM仍亟待研究。该文致力于对PTNMT的现状和相关问题进行系统性的整理和分析,从引入PTM的预训练方法、使用策略以及特定任务等角度对PTNMT方法进行详细的分类,并对PTNMT方法解决的问题进行总结,最后对PTNMT的研究进行展望。

Neural machine translation(NMT)models are usually trained using bilingual data.Building large-scale bilingual datasets is a huge challenge.In contrast,large-scale monolingual datasets for most languages are easier to construct.Pre-trained models(PTM)proposed in recent years can be trained on massive monolingual data.The generic representation of knowledge learned through pre-training helps achieve significant performance gains in downstream tasks.Currently pre-trained neural machine translation(PTNMT)has been extensively validated on resource-constrained datasets,but how to efficiently utilize PTM on high-resource NMT remains to be discussed.This paper focuses on reviewing and analyzing the current state and related problems of PTNMT,classifing PTNMT methods in terms of PTM's pre-trained methods,strategies,or specific tasks.We summarize the problems solved by PTNMT's methods,and conclude with a future outlook on PTNMT research.