云计算环境中HDFS数据块存储策略研究

Research on Placement Strategy of HDFS Data Blocks in Cloud Computing Environment

HDFS(Hadoop Distributed File System)以流式数据访问模式存储超大文件,具有高可靠性、高扩展性、低成本等特性,已广泛运行于商用硬件集群中。但在云计算系统中,由于采用了虚拟化技术,文件存储时如采用HDFS默认的存储策略,将带来数据可靠性的下降。本文通过对HDFS存储方法的改进,提出了一种充分考虑云环境中虚拟机存储位置的数据块存储策略,避免了多个数据块副本存储在同一台物理机器上。实验结果证明,该方法均衡了数据块在物理节点中的存储,提高了系统的可靠性。

HDFS(Hadoop Distributed File System),which stores large files in streaming data access mode,has the characteristics of high reliability,high scalability and low cost,and has been widely used in commercial hardware clusters.However,in the cloud computing system,due to the virtualization technology,if the default storage strategy of HDFS is used in file storage,the data reli⁃ability will be reduced.Through the improvement of HDFS storage method,this paper proposes a data block storage strategy that fully considers the storage location of virtual machine in cloud environment,avoiding multiple data block copies stored on the same physical machine.Experimental result shows that this method balances the placement of data blocks in physical nodes and im⁃proves the reliability of the system.