基于语料信息度量的文本分类性能影响研究

Study on the Influences of Text Categorization Performance Based on Corpus Information Measurement

基于不同的分类算法针对特性迥异的语料数据进行分类,其分类效果往往不同。通过研究分类算法针对专门语料库与自建语料库分类效果各不相同的根本原因,提出一种提高分类性能的新途径。从不同语料库的自动分类对比入手,定义类别聚类密度、类别复杂度、类别清晰度三个指标对语料库信息进行度量,通过多因素方差分析考察三个指标与分类性能的关系,得出语料的各项指标对不同分类算法分类性能的影响关系,并提出一种基于类别清晰度的交叠类文本分类方法以验证指标的有效性。实验表明:该三个指标都在不同程度上影响着分类算法的分类性能。语料类别的聚类密度越高,复杂度越低,类别清晰度越高,其表现出的分类效果越好。

The categorization performances usually vary in different corpus data with different categorization algorithms. The article proposes a new method to improve the categorization performance based on the analysis of the basic reason for the difference in categorization effects of the specialized corpus and the self-built corpus. It measures the corpus information from the comparison of the automatic catego-rization performances of different corpus through defining three indexes, namely, the category clustering density, the category complexity and the category definition. And it inspects the relationship between the three indexes and the categorization performance with multiple factors analysis of variance to obtain the effect relationship of the different indexes on the different algorithms categorization performances, and proposes an overlap text categorization method based on the category definition to verify the validity of the index. The experiments show that three indexes all affect the categorization performance of different algorithms to some extent. The higher clustering density, the lower complexity and the higher category definition, the better categorizationperformances will be.