藏文自动分词系统的设计

Design of a Tibetan Word Segmentation System

语料库作为基本的语言数据库和知识库,是各种自然语言处理方法实现的基础。随着统计方法在自然语言处理中的广泛应用,语料库建设已成为重要的研究课题。自动分词是句法分析的一项不可或缺的基础性工作,其性能直接影响句法分析。本文通过对85万字节藏语语料的统计分析和藏语词的分布特点、语法功能研究,介绍基于词典库的藏文自动分词系统的模型,给出了切分用词典库的结构、格分块算法和还原算法。系统的研制为藏文输入法研究、藏文电子词典建设、藏文字词频统计、搜索引擎的设计和实现、机器翻译系统的开发、网络信息安全、藏文语料库建设以及藏语语义分析研究奠定了基础。

As the fundamental linguistic knowledge base,human-annotated corpora are the basis of many statistical natural language processing tasks.Along with the wide use of statistical methods in natural language processing,corpus construction becomes an important research area.Word segmentation is necessary prerequisite of syntax parsing;its performance determines the parsing accuracy in a large degree.By the statistical analysis on a Tibetan corpus with 850,000 bytes,we first investigate the distribution and the syntactic function of Tibetan words,introduce a dictionary-based Tibetan word segmentation model,and then present the dictionary structure,case-auxiliary blocking and restoring algorithms which are necessary to Tibetan word segmentation.The development of the Tibetan word segmentation system also facilitates the research of the Tibetan word input methods,the Tibetan electronic dictionary construction,the Tibetan word frequency statistics,the design and realization of the search engine,the development of the machine translation system,the security of the network information,the construction of the Tibetan corpus,and the Tibetan semantic analysis.