串频统计和词形匹配相结合的汉语自动分词系统

An Chinese Word Automatic Segmentation System Based on String Frequency Statistics Combined with Word Matching

本文介绍了一种汉语自动分词软件系统，该系统对原文进行三遍扫描：第一遍，利用切分标记将文本切分成汉字短串的序列；第二遍，根据各短串的每个子串在上下文中的频度计算其权值，权值大的子串视为候选词；第三遍，利用候选词集和一部常用词词典对汉字短串进行切分。实验表明，该分词系统的分词精度在１．５％左右，能够识别大部分生词。

This paper presents a software system on Chinese automatic word segmentation.The original text is scanned three times:first,the text is cut into short Chinese character string sequence by cut marks;second,every short sting is weighted by its frequency in context,and the short strings weighted heavy are regarded as candidate words;third,short strings are segmented by candidate word set and everyday words.Experiments results shows that the segmentation precision of this word segmentation system is aboue 1.5%,and a large part of new words can be recognized correctly.This system is very suitable to document retrieval and other areas.