基于互信息和逻辑回归的新词发现

Extracting New Words with Mutual Information and Logistic Regression

【目的】改进新词提取方法,将提取的新词加入现有分词模型,提高医学文本分词准确率。【方法】在传统互信息模型基础上,得到字、字串的统计量,连同字串是否成词,一起作为特征,建立逻辑回归分类模型,设计了一种新词识别算法。【结果】在湘雅医院皮肤科电子病历文本数据上进行系列实验,与PMI、PMI^2和PMI^3相比,融入逻辑回归的PMI模型可以取得最高的新词提取准确率(0.803)。【局限】建立逻辑回归分类模型时,需要对训练集字串是否成词进行人工判断。【结论】融入逻辑回归的互信息分词模型可以更好地识别新词,降低词语误判,对于医学文本词的切分具有良好的应用价值。

[Objective]This paper modified the method for new word extraction,which are used to improve the performance of medical text segmentation models.[Methods]With the help of traditional mutual information model,we obtained the statistics of words and strings.Then,we established a logical regression classification model with these data,and built an algorithm for new word identification.[Results]A series of experiments were carried out on the texts of electronic medical records from Dermatology Department of Xiangya Hospital.Compared with PMI,PMI^2 and PMI^3,our model with logistic regression achieved the highest accuracy of new words extraction(0.803).[Limitations]To establish the logistic regression model for classification,we have to manually judge whether or not the training strings are words.[Conclusions]The proposed model and algorithm could effectively identify new words from medical records.