基于有效子串标注的中文分词

Effective Subsequence-Based Tagging for Chinese Word Segmentation

由于基于已切分语料的学习方法和体系的兴起,中文分词在本世纪的头几年取得了显著的突破。尤其是2003年国际中文分词评测活动Bakeoff开展以来,基于字标注的统计学习方法引起了广泛关注。本文探讨这一学习框架的推广问题,以一种更为可靠的算法寻找更长的标注单元来实现中文分词的大规模语料学习,同时改进已有工作的不足。我们提出子串标注的一般化框架,包括两个步骤,一是确定有效子串词典的迭代最大匹配过滤算法,二是在给定文本上实现子串单元识别的双词典最大匹配算法。该方法的有效性在Bakeoff-2005评测语料上获得了验证。

The research of automatic Chinese word segmentation has been advancing rapidly in recent years, especially after the First International Chinese Word Segmentation Bakeoff held in 2003. In particular, character-based tagging has claimed a great success in this field. In this paper, we attempt to generalize this method to subsequencebased tagging. Our goal is to find longer tagging units through a reliable algorithm. We propose a two-step framework to serve this purpose. In the first step, an iterative maximum matching filtering algorithm is applied to obtain an effective subsequence lexicon, while in the second step, a bi-lexicon based maximum matching algorithm is employed for identifying subsequence units. The effectiveness of this approach is verified by our experiments using two closed test data sets from Bakeoff-2005.