基于频繁词集聚类的海量短文分类方法

Massive short documents classification method based on frequent term set clustering

信息技术的飞速发展造成了大量的文本数据累积,其中很大一部分是短文本数据。文本分类技术对于从这些海量短文中自动获取知识具有重要意义。但是对于关键词出现次数少的短文,现有的一般文本挖掘算法很难得到可接受的准确度。一些基于语义的分类方法获得了较好的准确度但又由于其低效性而无法适用于海量数据。针对这个问题提出了一个新颖的基于频繁词集聚类的短文分类算法。该算法使用频繁词集聚类来压缩数据,并使用语义信息进行分类。实验表明该算法在对海量短文进行分类时,其准确度和性能超过其它的算法。

With the rapid development of information technology, huge data is accumulated. A vast amount of such data appears as short documents. It is very useful to classify such short documents to get knowledge automatically form the data. But most of the current classification algorithms can not get acceptable accuracy since key words appear few times in short documents. Some classification algorithms based on semantic information are fnore accurate but they are inefficient to be used to process very large document sets. A novel classification method based on frequent term set clustering is proposed. This method uses frequent term set clustering to compress massive data and uses semantic information to improve accuracy. Experimental study shows that this method is more accurate and efficient than other classification algorithms when classifying massive short documents.