摘要
Automatic word-segmentation is widely used in the ambiguity cancellation when processing large-scale real text,but during the process of unknown word detection in Chinese word segmentation,many detected word candidates are invalid.These false unknown word candidates deteriorate the overall segmentation accuracy,as it will affect the segmentation accuracy of known words.In this paper,we propose several methods for reducing the difficulties and improving the accuracy of the word-segmentation of written Chinese,such as full segmentation of a sentence,processing the duplicative word,idioms and statistical identification for unknown words.A simulation shows the feasibility of our proposed methods in improving the accuracy of word-segmentation of Chinese.
Automatic word- segmentation is widely used in the ambiguity cancellation when processing large- scale real text,but during the process of unknown word detection in Chinese word segmentation,many detected word candidates are invalid. These false unknown word candidates deteriorate the overall segmentation accuracy,as it will affect the segmentation accuracy of known words. In this paper, we propose several methods for reducing the difficulties and improving the accuracy of the word- segmentation of written Chinese, such as full segmentation of a sentence, processing the duplicative word, idioms and statistical identification for unknown words. A simulation shows the feasibility of our proposed methods in improving the accuracy of word - segmentation of Chinese.
出处
《现代电子技术》
2007年第4期108-110,共3页
Modern Electronics Technique
关键词
中文信息处理
汉字处理
自动分割
效率分析
Chinese information processing
Chinese automatic word-segmentation
ambiguity cancellation