MLWS2017中藏文分词评测的技术报告

Technical report on evaluation of Tibetan words in MLWS2017

随着语言信息处理数据规模和自动分析处理需求的增长,以及国内外学术交流的不断增加,自然语言信息处理技术的公共评测对研究的促进作用已在业界形成共识。词法分析是语言信息处理的基础与关键。文章针对MLWS2017中有关藏文的情况介绍了藏文分词评测语料的收集、整理情况,分析了藏文新闻类语料计算机分词的难度;对藏文分词评测分析软件设计思想进行研究基础上,设计了藏文评测分析软件,对软件进行了各项测试;分析藏文分词评测的结果,并验证了评测结果的正确性。

With the analysis processing, evaluation of natural increase of the scale of language information processing data and the demand of automatic and the increasing of academic exchange at home and abroad, the promotion of the public language information processing technology has formed a consensus in the Academic community. Lexical analysis is the basis and key to the language information processing. National Language and Information Processing Committee and Computational Linguistics Committee of Chinese Information Processing Society of China jointly organized the Evaluation of ＂Minority Language Word Segmentation （MLWS2017）＂, which is an evaluation word segmentation technology of text in Mongolian, Tibetan, and Uygur news. In this paper we mainly focused on the collection and collation of Tibetan word segmentation in MLWS2017, and introduced and analyzed the difficulty of computer word segmentation of text in Tibetan news. On the basis of study on the idea of design Tibetan word segmentation evaluation software, we designed an analysis software of Tibetan word evaluation. Furthermore, the software was tested using different kinds of methods to ensure its correctness and the correctness of the evaluation results is verified by other methods.