基于Hadoop的访问热点副本迁移技术

Replication Migration Technology for Access Hotspots Based on Hadoop

提出一种云环境下的访问热点负载均衡模型:基于节点的吞吐量与响应时间等主要参考指标,构建节点负载判定模块;文件在HDFS存储的过程中,将文件对应的数据块编号与存储路径相结合,设计存放在数据节点中的数据块到文件目录映射表;提出一种基于节点负载以及节点的存储空间的迁移源节点和目标节点选择方法;基于机架感知的机制,制定一种动态副本迁移方案。最后利用执行器下发指令给相应的数据节点,执行具体的迁移任务以及完善迁移后副本因子等参数信息的调整。通过迅速扩散副本的方式,来增加热点文件的副本数量,使得系统能够对外提供更大的吞吐量,缩短系统反应时间。

This paper proposes an access hotspots load balance model of cloud environment. Based on the node throughput and response time of the main reference index, this paper constructs the node load determination module. In the process of files sto- ring to HDFS, combining the file corresponding to the data block number with storage paths, the blocks of data stored in the data node are designed to a file directory mapping table. A selection method based on the node load and the storage space of migration of source node and destination node is puts forward. Based on the frame of perception mechanism, a dynamic copy migration scheme is established. Finally using the actuator issued instructions to the appropriate data nodes, it carries on the specific migra- tion tasks and improves the migrated copy factor to adjust parameters, such as information. This paper, by means of copy quickly spread to increase hot file copy number, enables the system to provide greater throughput and shortens the system response time.