Nutch中文分词的设计与实现

Design and Implementation of Chinese Word Segmentation in Nutch

面对与日俱增的中文信息检索需求,Nutch作为一个开源的搜索引擎系统平台受到众多开发者的青睐.但由于Nutch是基于英文的系统,不具备中文分词能力,因此,研究中文分词方法在Nutch中的应用对实现中文搜索引擎具有非常现实的意义.在对中文分词技术进行研究的基础上,设计并实现了具有中文分词功能和新词识别功能的分词器,在Nutch中实现了中文分词功能.实验测试结果表明,算法的分词效果能够达到预期的中文分词的要求.

Faced with growing demand of Chinese information retrieval, Nutch as a platform of open source search engine is favored by many developers. Nutch is based on the English system, without the capacity of Chinese word segmentation, so it has a very practical significance for realizing the Chinese search engine to research Chinese words in the application of Nutch. Based on the study of Chinese word segmentation, a word-dividing machine with the function of Chinese word segmentation and recognizing new word, is designed and implemented for the function of Chinese word segmentation in Nutch. Experi- mental results show that the effect of the segmentation algorithm can achieve the desired Chinese word re- quirements.