贫语言资源条件下藏文分句数据集构建研究

Study on the Construction of Tibetan Sentence Segmentation Dataset under Scarcity Language Resources

基于深度学习的藏文自动分句研究中构建分句数据集,事关藏文分句模型性能和质量。鉴于现有的藏文自动分句数据稀缺问题,文章通过梳理藏文句法结构,提出了位于句末的谓语动词和谓语形容词,以及终结虚词和离合虚词可充当句尾标识符号的观点,并通过相关语料库构建了谓语形容词词典、谓语动词词典和句尾虚词词典,最终使用句尾词匹对方法成功从语料中切分出了40万条句子,解决了藏文分句数据集建构问题,为基于深度学习的藏文分句研究提供了可靠和较大规模的数据基础。

The construction of a clause dataset in the study of Tibetan automatic sentence segmentation based on deep learning is related to the performance and quality of the Tibetan sentence segmentation model.In view of the scarcity of existing Tibetan automatic clause segmentation data,by combing various syntactic structures of Tibetan,in this paper we propose that predicate verbs and predicate adjectives at the end of sentences,as well as terminal function words and clutch function words,can be used as the end-of-sentence identifiers,and then constructs predicate adjective dictionary,predicate verb dictionary and end-of-sentence function word dictionary through the relevant corpus,and finally successfully cuts 400000 sentences from the corpus by using the endof-sentence matching method,which solves the problem of constructing Tibetan clause datasets.It provides a reliable and large-scale data basis for Tibetan sentence segmentation study based on deep learning.