隐马尔可夫模型的优化及其用于多文本实体识别

Research on Multi-Text Entity Recognition Based on Optimized Hidden Markov Model

针对传统HMM模型存在的上下文信息获取困难、未登录词无法处理等问题,本文提出一种优化的HMM模型。优化后的模型充分考虑了上下文的语义联系和依赖关系,采用Bi-gram指数线性插值算法,消除零概率事件,并对未登录词进行处理。使用改进的Viterbi算法求解最可能的状态序列并输出结果,提高模型的识别效果。使用简历数据集和CCKS2017电子病历数据集进行模型对比验证,实验结果表明,优化的HMM模型的实体识别效果优于传统的HMM模型,在CCKS2017电子病历数据集中的准确率和F1值分别达到91.61%和91.21%,提升了15.84%和11.78%;在简历数据集中的准确率和F1值分别达到91.29%和91.07%,提升了8.67%和6.88%。

The traditional HMM model has problems such as difficulty in obtaining context information and inability to deal with unknown words.Therefore,an optimized HMM model is proposed.The optimized model takes full account of the semantic relations and dependencies of the context.The Bi-gram exponential linear interpolation algorithm is used to eliminate zero probability events,and the unknown words are processed.The improved Viterbi algorithm is used to solve the most possible state sequence and output the results to improve the recognition effect of the model.Using the resume dataset and CCKS2017 medical dataset for model comparison and verification,the experimental results show that the entity recognition effect of the optimized HMM model is better than that of the traditional HMM model.The accuracy and F1 value in the CCKS2017 electronic medical record dataset reached 91.61%and 91.21%respectively,increasing by 15.84%and 11.78%.The accuracy and F1 value in the resume dataset reached 91.29%and 91.07%respectively,an increase of 8.67%and 6.88%.