基于数据增强的藏文改写检测研究

Tibetan Paraphrase Detection Based on Data Augment

该文针对藏文语料稀缺的问题,在藏汉双语、藏文单语文本改写检测任务中使用数据增强的方法,在一定程度上解决了低资源语言训练语料规模小的问题。在藏汉跨语言文本改写检测任务中,该文使用数据增强方法,有效利用目前公开的藏汉平行语料,扩充藏汉跨语言文本改写检测训练语料,当扩充至20万句对时,藏汉改写检测模型的皮尔森系数(pearson correlation)达到0.5476,比基线系统的皮尔森系数提升了0.3971,表明藏汉改写检测模型检测出的句对相似度值与人工标注的相似度值已达到中等程度相关。在藏文单语言任务中,该文采用训练藏文音节向量的方法,以缓解语料稀缺带来的词向量稀疏问题。实验结果表明,基于藏文音节向量的藏文改写检测模型的皮尔森系数可达到0.6780,比相应的基于藏文词向量实验的结果提升了0.1,使得藏文单语言文本改写检测模型的检测结果与人工标注的结果达到了强相关程度。

To alleviate the scarcity of Tibetan language corpus,this paper proposed data augment method for Tibetan-Chinese bilingual paraphrase and Tibetan paraphrase detection.In Tibetan-Chinese bilingual paraphrase detection task,this paper proposed to augment the parallel corpora available by the Tibetan monolingual texts.When the training set is expanded to 200,000 pairs,the Pearson coefficient of the experiment is increased from 0.3971 to 0.5476 for the baseline system.In Tibetan text paraphrasing detection task,Tibetan syllable vectors is adopted to alleviate the insufficient training corpus for the word vector.Experimental results show that the Pearson correlation based on the Tibetan syllable vector experiment reaches 0.6780,which is 0.1 higher than the corresponding word vector based method.