使用特征词的统计分布信息进行文本分类

Utilizing Term Statistical Distribution Information on Document Classification

文本分类是文本信息处理工作中的一个重要预处理部分。对常用的文本表示模型-向量空间模型的特征词权重估计方法提出了质疑,指出该方法的两个假设前提在一些情况下是不完全正确的,并详细分析了造成这一问题的原因。在此基础上,提出了正确估计特征词权重的三个基本假设,并由此得出新的特征词权重估计方法和文本分类方式,最大限度地利用了文本信息。

Document classification is a important preprocess work in intormation task To the ordmary document describable model-the Vector Space Model,the paper brings forward the problem about its term weight estimate measure,and points out that two hypothesis precondition of this way is not always right under some instances.We clearly analysis the reason which cause this problem.Based on that,the paper presents three basal hypothesis to estimate the term weight rightly,and use them to get new term weight estimate measure and document classified algorithm,utilize the document information in the most extent.