Web平行语料挖掘及其在机器翻译中的应用

Mining Parallel Corpora from Web and Its Application in Machine Translation

双语平行语料库在自然语言处理领域有很多重要应用,但是大规模双语平行语料库的自动获取并不容易。该文提出了一种有效的从Web上获取高质量双语平行语料库的方案,研究了候选双语混合网页获取和平行句对抽取等关键技术。运用该文方法共获取了258万双语平行句对,平均正确率为93.75%,其中前150万句对的平均正确率达到96%。该文还提出句对质量排序和领域信息检索两种方法将Web数据应用于统计机器翻译的模型训练,在IWSLT评测数据上BLEU值可以提高2到5个百分点。

Bilingual parallel corpora can be used in many applications of NI.P, but it＇s not easy to acquire the large scale corpora automatically. This paper proposes an effective solution to mine high-quality bilingual parallel corpora from web pages and analyses the key technology of obtaining eandidate mix-languages web pages and sentence align- ment. We have extracted 1.67 million parallel sentences, whieh average accuracy is 93.75%, and the accuracy of the first 1 million sentences is 96%. This paper also proposes the sentences re ranking method and domain informa tion retrieval method to apply the web data to the training of SMT model. Experiments conducted on the IWSLT tasks show 2 to 5 BI.EU gains over baseline.