基于农业垂直搜索引擎中文分词词典的构建研究

Construction of Chinese word segmentation dictionary based on agricultural vertical search engine

在农业垂直搜索引擎研究过程中,中文分词是重要的研究方向。针对传统农业垂直搜索引擎搜索信息抽取不准确、速度慢等缺点,采用双数组Trie树为基本模型,利用中文词条首字区位码与数据库表行号相对应的方式,并根据农业垂直搜索引擎的需要设置了农业词汇的词性编码,以My SQL数据库为例设计了农业领域专用的分词词典。该分词词典可充分利用数据库的优势进行词典组织,并且可以进行词库的远程共享和共同维护,方便不同的系统进行访问;词条按首字分类存放构造双数组Trie树,可有效减少构造过程的内存空间。该农业分词词典结构对其他领域和行业也具有借鉴意义。

In the process of agricultural vertical search engine research, Chinese word segmentation is an important research direction. Vertical search engines existed inaccuracy, slow velocity and other shortcomings for information extraction based on traditional agricultural. In this paper, the Trie tree method was adopted as the basic model to design the word segmentation dictionary specifically for agricultural use based on MySQL database. The word segmentation dictionary could make full use of the database for dictionary. It could he a thesaurus remote sharing and common maintenance, convenient access to different system. In the dictionary, used the term in Chinese location code and database table row number corresponding to the acronym, and according to the needs of agricultural vertical search engine, set up agricultural word part of speech coding. This dictionary stored the double array Trie tree according to the classification of storage structure. It could reduce the memory space of construction effectively. At the same time, the agricultural word segmentation dictionary structure also had reference significance to other field.