基于跨语种预训练语言模型XLM-R的神经机器翻译方法

Neural Machine Translation Based on XLM-R Cross-lingual Pre-training Language Model

探索将XLM-R跨语种预训练语言模型应用在神经机器翻译的源语言端、目标语言端和两端,提高机器翻译的质量。提出3种网络模型,分别在Transformer神经网络模型的编码器、解码器以及两端同时引入预训练的XLM-R多语种词语表示。在WMT英语-德语、IWSLT英语-葡萄牙语以及英语-越南语等翻译中的实验结果表明,对双语平行语料资源丰富的翻译任务,引入XLM-R可以很好地对源语言句子进行编码,从而提高翻译质量;对双语平行语料资源匮乏的翻译任务,引入XLM-R不仅可以很好地对源语言句子进行编码,还可以对源语言端和目标语言端的知识同时进行补充,提高翻译质量。

The authors explore the application of XLM-R cross-lingual pre-training language model into the source language,into the target language and into both of them to improve the quality of machine translation,and propose three neural network models,which integrate pre-trained XLM-R multilingual word representation into the Transformer encoder,into the Transformer decoder and into both of them respectively.The experimental results on WMT English-German,IWSLT English-Portuguese and English-Vietnamese machine translation benchmarks show that integrating XLM-R model into Transformer encoder can effectively encode the source sentences and improve the system performance for resource-rich translation task.For resource-poor translation task,integrating XLM-R model can not only encode the source sentences well,but also supplement the source language knowledge and target language knowledge at the same time,thus improve the translation quality.