神经机器翻译的词级别正则化

Word level regularization for neural machine translation

由于神经机器翻译模型具有大规模参数且其性能极大地依赖大规模高质量平行语料,当训练数据规模小于模型复杂度时,模型易出现过拟合问题而泛化能力不足.针对此问题,研究了词级别的正则化技术,通过对模型输入句子中的单词进行随机干扰,以此削弱数据的特异性,从而抑制模型对于数据的过度学习,防止过拟合,提高模型的泛化能力.通过Transformer模型在标准规模中文-英语数据集和中小规模英语-土耳其语数据集上进行的相关实验,结果证明词级别正则化方法能够使模型在收敛后更加稳定,不易出现过拟合的情况,并且翻译质量也得到了明显的提升.

For the reason that neural machine translation(NMT)model have large-scale parameters and its performance relys on large-scale and high-quality parallel corpus,the model is prone to over-fitting and insufficient generalization ability while the training data size is smaller than the model complexity.To resolve this problem,we study the word-level regularization technology,which reduces the specificity of the data by randomly perturbing the words in the input sentence of the model,thereby inhibiting the model from over-learning,preventing the data over-fitting,and improving the generalization ability of the model.Through the relevant experiments of the Transformer model on the standard-scale Chinese-English dataset and the small-scale English-Turkish dataset,our results prove that the word-level regularization method can stabilize the model after the convergence and is not prone to over-fitting.Finally,the translation quality has also been significantly improved.