基于语义的文档特征提取研究方法

Semantic-based Feature Extraction Method for Document

中文文本特征词选取是文本处理的重要方面,对文本分类有重要影响。现有的文本特征提取方法存在生成特征向量维数高、依赖训练集、忽略低频关键词等不足。利用《同义词词林》计算词语之间的语义距离,通过聚类算法筛选类别的主题相关词,最后通过信息增益算法从主题相关词中选取特征词。以宏F值和微F值为评价指标,通过有效性实验和对比实验表明,该方法的文本特征选取效果优于其他经典算法。

Feature extraction of Chinese documents is an important part in the document processing,and imposes great influence on the document classification.Pre-existing document feature extraction methods have many shortcomings,such as creating a feature vector of high dimensions,depending on training sets,ignoring low-frequency keywords,and so on.In this paper,the semantic distance between words was calculated based on the synonyms dictionary,and then theme related words of each classification were selected by the density clustering method,and finally the feature words were selected from the theme related words using the information gain algorithm.In order to validate the proposed method,one validation experiment and one comparison experiment were designed and the evaluation indexes including the macro-F value and the micro-F value were calculated.Experiment results show that the proposed document feature extraction method has better performance than other traditional algorithms.