基于调度树的分布式文件系统数据流调度算法研究

Research on data stream scheduling algorithm of distributed file system based on scheduling tree

文章针对HDFS的数据写入性能不高的问题提出了一种基于调度树的数据流并发传输的调度算法。该算法将文件系统的节点分为转发节点和叶子节点,按流量匹配原则将节点编排为调度树后进行副本分发,让所有节点的网卡并发传输,磁盘并发写入,减小了副本写入分布式文件系统的时间。性能测试表明,使用该算法的分布式文件系统的写入性能较原始的HDFS更高。

This article proposes a scheduling algorithm for concurrent transmission of data flow based on scheduling tree with an aim to improve the performance of HDFS.The algorithm divides the nodes of the file system into forwarding nodes and leaf nodes,distributes the duplicates by arranging the nodes into the scheduling tree according to the principle of information flow matching,and realize the concurrent transmission of adapters and the concurrent disk input which shortens the time used for writing the duplicates into the distributed files.The performance tests shows that the distributed file system of this algorithm is better than the original HDFS with regard to disk input.