藏文词库自扩充学习算法的设计与实现

A design and an implementation of learning algorithm for the Tibetan Lexicon

在自然语言处理过程中,词库的大小直接影响着快速输入、信息检索和文本相似度对比等一系列工作的准确性和效率。目前,藏文词库建设主要靠人工扩充方法,未见藏文词库学习算法的研究或文献。文章设计了一种通过遍历藏文文本音节字并识别和统计每个词,将达到阈值的词添加到藏文词库中的算法。经测试,该算法在词库扩充方面具有良好的效果。

In the process of the natural language,the size of thesaurus directly affects the accuracy and efficien?cy of a series of work,such as fast input,information retrieval and text similarity comparison.At present,the Ti?betan lexicon mainly depends on manual construction and expansion,and researches or literatures on Tibetan lexicon learning algorithm were not seen.In this paper,an algorithm is designed to add the words that reach the threshold to the Tibetan lexicon by traversing the syllables of the Tibetan text,recognizing and counting each word.The test results showed that the algorithm has a good effect in the expansion of the vocabulary.