向量空间法中单词权重函数的分析和构造

ANALYSIS AND CONSTRUCTION OF WORD WEIGHING FUNCTION IN VSM

文本分类是文本挖掘的基础与核心 ,是近年来数据挖掘和网络挖掘的一个研究热点 ,在传统的情报检索、网站索引体系结构的建立和 Web信息检索等方面占有重要地位 .深入分析了一种简单而常用的经典文本分类模型——向量空间模型 ( vector space model,VSM)——的实质 ,找出了其分类精度低的原因 ,提出了一种利用特征筛选中的评估函数代替 IDF函数进行权值调整的方法 ,并对采用各种不同评估函数进行权值调整的性能进行了理论分析和实验比较 ,提出了一种构造新的高性能评估函数的新颖方法 .

Text classification is the basis and core of text mining, and plays an important rule in traditional information retrieval, construction of website architecture, and search for web information. It has become a hot research project in recent years. In this paper, the hypostasis of VSM (vector space model), a kind of frequently-used classical text classification model, is analyzed to find the reason for its low classification precision, and a weight adjustment method is put forward in which the IDF function is replaced by evaluation function used in feature selection. Also made are theoretic analysis and experimental comparison with the performance of weight adjustment using various evaluation functions. And a novel approach to construct a new high-powered evaluation function is presented.