Automatic word-segmentation is widely used in the ambiguity cancellation when processing large-scale real text,but during the process of unknown word detection in Chinese word segmentation,many detected word candidate...Automatic word-segmentation is widely used in the ambiguity cancellation when processing large-scale real text,but during the process of unknown word detection in Chinese word segmentation,many detected word candidates are invalid.These false unknown word candidates deteriorate the overall segmentation accuracy,as it will affect the segmentation accuracy of known words.In this paper,we propose several methods for reducing the difficulties and improving the accuracy of the word-segmentation of written Chinese,such as full segmentation of a sentence,processing the duplicative word,idioms and statistical identification for unknown words.A simulation shows the feasibility of our proposed methods in improving the accuracy of word-segmentation of Chinese.展开更多
中文自动分词技术在中文信息处理、Web文档挖掘等处理文档类研究中是一项关键技术,而分词算法是其中的核心。正向最大匹配算法FMM(Forward Maximum Match)具有切分速度快、简洁、容易实现等优点,但是还存在分词过程中设定的最大词长初...中文自动分词技术在中文信息处理、Web文档挖掘等处理文档类研究中是一项关键技术,而分词算法是其中的核心。正向最大匹配算法FMM(Forward Maximum Match)具有切分速度快、简洁、容易实现等优点,但是还存在分词过程中设定的最大词长初始值固定不变的问题,带来匹配次数相对较多的弊端。针对此问题提出了根据中文词典中的词条长度动态确定截取待处理文本长度的思想,改进了FMM算法,并用互信息统计来消除交集型歧义。最后,通过实验对算法进行了分词和验证,结果表明改进的算法与一般正向最大匹配算法相比,中文分词的准确率提高了。展开更多
文摘Automatic word-segmentation is widely used in the ambiguity cancellation when processing large-scale real text,but during the process of unknown word detection in Chinese word segmentation,many detected word candidates are invalid.These false unknown word candidates deteriorate the overall segmentation accuracy,as it will affect the segmentation accuracy of known words.In this paper,we propose several methods for reducing the difficulties and improving the accuracy of the word-segmentation of written Chinese,such as full segmentation of a sentence,processing the duplicative word,idioms and statistical identification for unknown words.A simulation shows the feasibility of our proposed methods in improving the accuracy of word-segmentation of Chinese.
文摘中文自动分词技术在中文信息处理、Web文档挖掘等处理文档类研究中是一项关键技术,而分词算法是其中的核心。正向最大匹配算法FMM(Forward Maximum Match)具有切分速度快、简洁、容易实现等优点,但是还存在分词过程中设定的最大词长初始值固定不变的问题,带来匹配次数相对较多的弊端。针对此问题提出了根据中文词典中的词条长度动态确定截取待处理文本长度的思想,改进了FMM算法,并用互信息统计来消除交集型歧义。最后,通过实验对算法进行了分词和验证,结果表明改进的算法与一般正向最大匹配算法相比,中文分词的准确率提高了。
