基于层叠隐马模型的汉语词法分析

Chinese Lexical Analysis Using Cascaded Hidden Markov Model

提出了一种基于层叠隐马模型的汉语词法分析方法 ,旨在将汉语分词、词性标注、切分排歧和未登录词识别集成到一个完整的理论框架中 在分词方面 ,采取的是基于类的隐马模型 ,在这层隐马模型中 ,未登录词和词典中收录的普通词一样处理 未登录词识别引入了角色HMM :Viterbi算法标注出全局最优的角色序列 ,然后在角色序列的基础上 ,识别出未登录词 ,并计算出真实的可信度 在切分排歧方面 ,提出了一种基于N 最短路径的策略 ,即 :在早期阶段召回N个最佳结果作为候选集 ,目的是覆盖尽可能多的歧义字段 ,最终的结果会在未登录词识别和词性标注之后 ,从N个最有潜力的候选结果中选优得到 不同层面的实验表明 ,层叠隐马模型的各个层面对汉语词法分析都发挥了积极的作用 实现了基于层叠隐马模型的汉语词法分析系统ICTCLAS ,该系统在 2 0 0 2年的“九七三”专家组评测中获得第 1名 ,在 2 0 0 3年汉语特别兴趣研究组 (ACLSpecialInterestGrouponChineseLanguageProcessing ,SIGHAN)组织的第 1届国际汉语分词大赛中综合得分获得两项第 1名、一项第 2名 这表明 :ICTCLAS是目前最好的汉语词法分析系统之一 。

This paper presents an approach for Chinese lexical analysis using cascaded hidden Markov model (CHMM), which aims to incorporate Chinese word segmentation, part-of-speech tagging, disambiguation and unknown words recognition into an integrated theoretical frame. A class-based HMM is applied in word segmentation, and in this model, unknown words are treated in the same way as common words listed in the lexicon. Unknown words are recognized with reliability on roles sequence tagged using Viterbi algorithm in roles HMM. As for disambiguation, the authors bring forth an n-shortest-path strategy that, in the early stage, reserves the top N segmentation results as candidates and covers more ambiguity. Various experiments show that each level in the CHMM contributes to Chinese lexical analysis. A CHMM-based system ICTCLAS is accomplished. The system ranked top in the official open evaluation, which was held by the “973” project in 2002. And ICTCLAS achieved 2 first ranks and 1 second rank in the first international word segmentation bakeoff held by SIGHAN (the ACL Special Interest Group on Chinese Language Processing) in 2003. It indicates that ICTCLAS is one of the best Chinese lexical analyzers. In a word, CHMM is effective for Chinese lexical analysis.