汉语交集型歧义切分字段关于专业领域的统计特性

Statistical Properties of Overlapping Word Segmentation Ambiguities in Domain-specific Chinese Corpora

交集型分词歧义是汉语自动分词中的主要歧义类型之一。现有的汉语自动分词系统对它的处理能力尚不能完全令人满意。针对交集型分词歧义,基于通用语料库的考察目前已有不少,但还没有基于专业领域语料库的相关考察。根据一个中等规模的汉语通用词表、一个规模约为9亿字的通用语料库和两个涵盖55个专业领域、总规模约为1.4亿字的专业领域语料库,对从通用语料库中抽取的高频交集型歧义切分字段在专业领域语料库中的统计特性,以及从专业领域语料库中抽取的交集型歧义切分字段关于专业领域的统计特性进行了穷尽式、多角度的考察。给出的观察结果对设计面向专业领域的汉语自动分词算法具有一定的参考价值。

Overlapping ambiguity is a major type of ambiguity in Chinese word segmentation. The performance of existing word segmentation systems in resolving this ambiguity is still unsatisfactory, especially in dealing with domain-specific texts. In contrast to those quite detailed statistical observations on overlapping ambiguities in generalpurpose corpus, similar observations in domain-specific corpus have not been reported in the literature. In terms of a medium-sized general-purpose Chinese wordlist, a general-purpose corpus with over 900 million characters and two domain-specific corpora with total 140 million characters covering 55 domains, statistical properties of high frequent overlapping ambiguities are addressed and studied from several perspectives： with overlapping ambiguity string from general corpus examined in the domain corpus, and vice versa. It is believed that the finding of this paper will benefit word segmentation disambiguation in particular for domain-specific texts.