面向藏汉神经机器翻译的数据筛选方法

Data Filtering Strategies for Tibetan-Chinese Neural Machine Translation

针对目前在藏汉机器翻译中使用传统数据增强方法会导致数据的语法和语义损失等问题,本文在传统数据增强方法的基础上,提出将句子困惑度与语义相似度相结合的伪数据筛选方法,通过困惑度降低伪数据的语法错误率,同时通过语义相似度减少伪数据的语义偏差,以更好地缓解低资源下平行数据质量欠佳和稀缺等问题。本文使用伪数据筛选方法在藏汉、英汉2对双向语种上进行实验,结果比传统的数据增强方法的BLEU值分别提升了0.11、0.53、1.18、1.08。由此表明,本文提出的伪数据筛选方法有效地改善了翻译模型在语法和语义上的缺陷,从而增强了翻译系统的性能以及提升了翻译模型的泛化能力,验证了本文方法的有效性。

Data syntax and semantic losses arise in Tibetan-Chinese machine translation when traditional data augmentation methods are employed.To address this issue,this paper proposes a pseudo-data filtering method combining sentence confusion degree with semantic similarity degree on the basis of traditional data enhancement methods.This strategy effectively tackles chal‐lenges such as the inadequate quality and scarcity of parallel data,particularly in low-resource settings.The results of this study demonstrate that the pseudo data filtering approach significantly improves both Tibetan-Chinese and English-Chinese bidirec‐tional language translation tasks.The proposed pseudo-data filtering method effectively improves the grammatical and semantic defects of the translation model,thus enhancing the performance of the translation system and the generalization ability of the translation model,and verifies the effectiveness of the proposed method.