中文分词中组合型切分歧义的消解研究

Research on combinational ambiguity strings in Chinese word segmentation

针对中文自动分词中组合型歧义消解难的问题,提出了一种新的切分算法来对组合型歧义字段进行消歧。该算法首先自动从训练语料中提取歧义字段的上下文信息来建立规则库,然后利用C-SVM模型结合规则对组合型歧义字段进行歧义消解。最后以1998年1月《人民日报》语料中出现的组合型歧义字段进行训练和测试,实验显示消歧的平均准确率达89.33%。

Combinational ambiguity is one of the most difficult problems in Chinese word segmentation.The paper discusses a new segmentation algorithm to solve the combinational ambiguity.The algorithm automatically extracts contextual information of the combinational ambiguity to establish rules,then uses the C-SVM model and these rules to solve the combinational ambiguity.The People Daily corpus of January 1998 is used in training and testing,and the average accuracy is 89.33%.