基于HDFS的大数据文件传输实验设计

Experimental Design of File Traction Based on HDFS Big Data

随着云计算应用技术及相关研究的发展,云编程模式也有了新的技术革新。在高校校园网实验教学体系中,云平台的相关实验已经成为大数据分析等课程配套实验的主要内容。使用HDFS架构,搭建稳定、实用、符合实验课程体系的大数据实验平台,成为高校校园网实验研究的新课题。选用开源云计算平台Hadoop,作为基础平台进行大数据分析实验,搭建基本的实验平台,进行数据处理平台的优化处理。通过HDFS为分布式计算存储提供底层应用支持,实现NameNode和DataNode的通信,将用户通过数据块的方式将文件存储到节点中,及时处理客户端的读写请求,并在NameNode的统一调度下进行数据块的创建、删除、复制和映射。根据具体的实验方法,有针对性的制定实验流程。

With the development of cloud computing application technology and related research,cloud programming model has also a new technological innovation.In college campus network experimental teaching system,cloud platform experiment has become main content for big data analysis.So,by using HDFS struck,building a stable,practical experiment platform to meet the experiment course system becomes a new topic of the experimental study of campus network in colleges and universities.In this paper,we use open source cloud computing platform Hadoop as the basic platform for big data analysis experiment.The basic experimental platform is used for data processing optimization.HDFS provides the underlying application support for distributed computing storage,and realizes communication between NameNode and DataNode.The user file is stored into the node through the data block,so that the read and write requests of the client can be processed timely,and the data block can be created,deleted,replicated and mapped with a unified schedule of NameNode.At the same time,we can target the experiment process according to experiment methods.