无监督分词算法在新词识别中的应用

Application of Unsupervised Word Segmentation Algorithm in New Word Recognition

新词识别过程中,使用分词工具进行预分词的方法,受限于训练语料而对某些领域的分词准确率不佳.针对这个问题,本文提出了一种改进方法.该方法首先基于元语言模型进行无监督预分词,再将词频、互信息和邻接熵作为主要特征进行新词发现.同时方法中还结合了命名实体识别对发现的结果进行过滤,得到候选词组后使用网格搜索寻找最优的超参数组合.实验选取四种不同领域的语料,在统一的超参数下,前10%的新词准确率分别达到了88.3%、80.5%、85.9%、91.9%.实验表明,这种无监督的分词方法适用于新词识别领域,并具备良好的领域适应性.

In new word recognition,the method which uses the common tools for word segmentation is not good in some fields because of the specific training corpus.This paper proposes an improved method for the problem.Firstly,we segment the word for an unsupervised method based on N-gram language model,and then use some features to discover new words including word frequency,mutual information and branch entropy.At the same time,the method also combines the named entity recognition to filter the results.And after obtaining the candidate words,the grid search method is used to find the optimal hyperparameter combination.We selected four different fields of corpus in the experiment.Under the same hyperparameters,the accuracy of the top 10%of new words reached 88.3%,80.5%,85.9%,and 91.9%,respectively.Experiments show that this unsupervised word segmentation method is available and has a good adaptability in the new word recognition.