词形还原方法及实现工具比较分析

Contrast Analysis of Methods and Tools for Lemmatization

结合理论和实验比较分析用于词形规范的词形还原方法和工具。归纳现有词形还原方法的主要分类,分析各类方法的特点和不足。介绍7种词形还原实现工具,并从其实现原理、使用的词性标注器、词典、开发语言、处理的语种、是否具有拼写检查功能等方面比较分析各工具的特点。选取其中5种工具,利用WordSimith Tools的标准数据进行词形还原实验。结合实验结果分析各工具的优劣,发现Specialist NLP Tools的词形还原工具具有较好的词形还原处理效果,为研究者选择适当的词形还原方法和工具提供参考。

Combining theory with practice, this paper compares the methods and tools for lemmatization in word normalization. It summarizes the categories of lemmatization methods and analyses their features and disadvantages. Then it separately compares seven tools from aspects as the principle, POS tagger, lexicon, programming language, language, spell checker. It takes experiments with the datasets from WordSimith Tools to evaluate five lemmatizers. By comparing the results, it finds that the Specialist NLP Tools has a better effect than others . This paper provides an assistance for the study in choosing the appropriate method and tool for lemmatization.