基于语料库文本自动分类算法及应用比较研究

A Comparative Study of Corpus Based Automatic Text Classification Algorithms

基于Python语言,利用公开中文语料库,测试不同算法模型对中文文本分类的效果。选择语料中不同数量的语料种类,首先对文本进行格式化读取、清洗等处理,而后以2∶1∶1的比例,分为训练集、验证集、测试集,最后依照文本表示、特征提取、分类算法选择、效果评估的步骤,依次在词袋、词嵌入、语言3种模型中选取典型代表进行中文文本分类。在深度学习模型的帮助下,文本分类得到了快速的发展,当前的主流分类方法基本都能满足不同任务的文本分类需求,特别是BERT语言模型可极大地提升文本分类的效果。

Based on Python,open Chinese corpus was used to test the effect of different algorithm models on Chinese text categorization.This paper selects different types of corpus,firstly formats,reads and cleans the text,and then divides it into training set,verification set and test set in the ratio of 2∶1∶1,and finally according to the steps of text representation,feature extraction,classification algorithm selection and effect evaluation,selects typical representatives from the three models of bag of words,word embedding and language Line Chinese text classification.With the help of deep learning model,text classification has developed rapidly.The current mainstream classification methods can basically meet the text classification requirements of different tasks,especially the BERT language model,which improves the effect of text classification to an unprecedented height.