基于免疫的中文网络短文本聚类算法

An Immune Based Algorithm for Chinese Network Short Text Clustering

网络短文本聚类是网络内容安全的一种主要处理方法.然而,中文网络短文本固有的关键词词频低、存在大量变形词等特点,使得难以直接使用现有面向长文本的聚类算法.本文提出了一种面向中文网络短文本的基于免疫网络调节的聚类算法.首先,利用抽取的中文词语的N-gram片段的拼音序列来组成一个中文网络短文本的特征表示,从而缓解关键词词频过低和存在变形词对聚类的影响;然后,将网络短文本集构建为一个动态网络,利用免疫网络学习机制来自动发现网络短文本之间的内在关联,获得合适的聚类结果.测试实验表明,相对于传统的聚类方法如K-means,本文的算法能够得到更好的中文网络短文本聚类效果.

Network short text clustering is a major technology in network content security. Since Chinese network short text is less of keywords and full of anomalous writings, the traditional text clustering method is not directly suitable for network short text clustering. This paper presents an immune network regulation based method to cluster Chinese network short texts. First, Chinese N-gram chunks are extracted and transformed to Chinese pinyin to form the feature representation to each Chinese network short text, so as to relieve these two characteristics＇ bad influence on the clustering performance. Then, the network short text set is constructed as a dynamic network and an immune network learning mechanism is used to learn the similarity among short texts and therefore to gain a better clustering result. Experiments show our method can get better performance in Chinese network short text clustering, compared with traditional method such as K-means.