高维数据的增量式聚类算法的距离度量选择研究

Selecting distance metrics for incremental clustering algorithm of high dimensional data

合适的距离度量函数对于聚类结果有重要的影响。针对大规模高维数据集,使用增量式聚类算法进行距离度量的选择分析。SpFCM算法是将大规模数据集分成小样本进行增量分批聚类,可在有限的计算机内存中获得较好的聚类结果。在传统的SpFCM算法的基础上,使用不同的距离度量函数来衡量样本之间的相似性,以得出不同的距离度量对SpFCM算法的影响。在不同的大规模高维数据集中,使用欧氏距离、余弦距离、相关系数距离和扩展的杰卡德距离来计算距离。实验结果表明,后3个距离度量相对于欧氏距离可以很大程度地提高聚类效果,其中相关系数距离可以得到较好的结果,余弦距离和扩展的杰卡德距离效果比较一般。

Appropriate distance metric functions have an important effect on clustering results.For large-scale and high-dimensional datasets,the incremental fuzzy clustering algorithm is used to analyze the selection of distance metrics.Since the SpFCM algorithm divides a large-scale dataset into small samples for incremental batch clustering,it can get better clustering results in limited computer memory.Different distance metric functions are applied into the traditional SpFCM algorithm in order to measure the similarities between different samples to check the effect of different distance metrics on the SpFCM algorithm.Four distance metrics,which are the Euclidean metric,the cosine metric,the correlation distance metric and the extended Jaccard similarity metric,are used to calculate the distance for different large-scale high dimensional datasets.Experimental results show that,the latter three distance metrics can greatly improve the clustering effect.The correlation distance metric gets a better clustering result while the cosine distance metric and the extended Jaccard similarity distance get an average result.