基于统计学和语义信息的中文文本主题识别技术

Topic detection technology for Chinese text based on statistics and semantic information

由于中文分词处理的复杂性在一定程度上限制了中文信息抽取技术的发展,因此,快速有效地抽取中文文本主题的需求越来越突出。该文主要通过中文分词技术、频繁词查找和词性组合计算来分析词与词之间的关联并最终提取出能够表达文章内容的主题词汇,同时还对这些词汇作了记分和排序。读者能够通过这些词汇来判定文章的主题和重要内容。通过对人民日报语料进行实验表明,该方法正确率能够保持在66%以上,同时对于网页邮件等真实文档也有较好的测试结果。

Requirements for extracting main information from Chinese texts sharply stand out because the complexity of Chinese word segments have partly restricted the development of Chinese information retrieval. A novel extraction method is proposed in this paper. The new method extracts the keywords and phrases expressing the main idea of text by using Chinese words segmentation, frequent searched words, and the parts of speech computation. Moreover, scoring and ordering of these extracted words are also given. The...