HDFS动态副本因子的优化研究

Research on Optimization of HDFS Dynamic Replication Factor

针对云环境中分布式文件系统(HDFS)静态副本机制存在的不足,提出一种改进的动态副本因子调整策略。该策略包含待调整副本因子文件筛选和文件副本因子动态调整两个部分,首先结合数据访问的时间局部性原理,根据文件访问热度定量描述获得待调整副本因子文件集合,通过设定两个不同长度的决策时间区间,针对不同访问热度以及不同决策时间区间内的文件分别采取不同的副本因子调整措施,做到性能和存储代价的有效折中。通过搭建分布式的实验环境验证动态副本因子调整策略的有效性,实验结果表明,随着文件访问热度的增加,改进策略可以有效降低系统作业平均响应时间,提升云环境下数据服务的性能。

We propose an improved dynamic replica factor adjustment strategy to solve the problem of HDFS static replica mechanism in cloud environment.It consists of two parts： file filtering of adjusted replica factor and dynamic adjustment of file copy factor.First,in combination of temporal-local principle of data access,according to the quantitative description of the file access heat,the set of the replica factor file is obtained,and by setting two decision time intervals of different lengths,different replica factor adjustment measures are taken for different access heat and files in different decision time intervals to achieve effective compromise between performance and storage cost.A distributed experimental environment is built to verify the validity of the proposed strategy,which shows that with the increase of the file access heat,the improved strategy can effectively reduce the average response time and improve the performance of the data service in the cloud environment.