结合大数据流特征和改进SOM聚类的资源动态分配算法

DYNAMIC RESOURCE ALLOCATION ALGORITHM BASED ON BIG DATA STREAM CHARACTERISTIC AND IMPROVED SOM CLUSTERING

在大数据流中,由于数据特征的未知性,如何分配数据资源是一个难题。为了解决这个问题,提出一种大数据环境下基于数据特征预测和改进自组织映射SOM(Self-organizing maps)的资源管理算法。根据数据的体积和速度变化,通过自回归模型对下一时间间隔到达的数据的特征进行估计,估计值用数据特征(CoD)向量表示;利用粒子群优化PSO算法来优化SOM算法的权重分布,形成改进型SOM算法,对CoD向量进行聚类,动态创建和分配云资源集群。这些集群以拓扑排序的方式创建,集群之间的联系越多,它们的排序越接近,利用这种拓扑排序来减少等待时间。实验结果表明,该算法能准确预测数据特征,有效提高了云资源的利用率。

In big data streams,it is a difficult problem to allocate data resources due to the unknown characteristics of data.To solve this problem,we proposed a resource management algorithm based on data characteristic prediction and improved self-organizing mapping(SOM)in big data environment.We estimated the data characteristics arriving at the next time interval by the autoregressive model according to the volume and velocity of the data,and the estimated value was represented by the characteristics of data(CoD)vector.Then we used the particle swarm optimization(PSO)to optimize the weight distribution of SOM algorithm,and an improved SOM algorithm was formed to cluster the CoD vectors,so as to dynamically create and allocate cloud resource clusters.These clusters were created in a topological sort.The more connections between clusters,the closer their sort was,so using this topological sort could reduce waiting time.The experimental results show that the proposed algorithm can not only accurately predict the data characteristics,but also effectively improve the utilization of cloud resources.