纠删码存储下的离线批处理作业性能优化

Performance Optimization of Offline Batch Jobs in Erasure-Coded Storage Systems

随着互联网数据的爆发式增长,越来越多的分布式存储系统开始引入纠删码存储机制,以在提供数据可靠性的同时降低存储开销。但纠删码机制的引入改变了数据放置模式,从而影响分布式系统上层业务的数据访问和运行效率。在异构Hadoop集群环境中,一类典型的离线批处理作业——MapReduce应用在条带式纠删码存储模式下需要从多个节点访问数据,该“一对多”的数据访问模式由于节点性能差异造成应用执行效率下降。对此,该文提出了一种基于异构环境的数据放置和任务分配策略。通过对异构集群中各节点的硬件参数和历史负载进行分析,将同一纠删码条带的数据块尽可能分布在性能相近的节点上;在系统进行任务分配时,针对各节点当前负载和运算能力确定节点的任务并发度,以平衡各节点计算资源的占用情况,从而避免因数据访问或计算过程中的资源竞争产生极端缓慢任务以致降低整个MapReduce应用的运行效率。实验结果表明,相比当前Hadoop默认的随机数据放置和任务分配策略,该文提出的异构感知数据放置策略和动态任务分配策略能够在不同类型的MapReduce应用中有效削弱任务的长尾效应,使得作业整体运行时间节约10.5%~42%,验证了该方案的有效性。

With the explosive growth of Internet data, many distributed storage systems have integrated erasure-coding mechanisms to ensure data reliability, while further reducing storage overhead. However,erasure-coding has changed the data placement scheme, thus affecting the data access of other services of the cluster. This paper proposes a new data placement scheme and a task scheduling strategy based on heterogeneous Hadoop cluster that can be better adapted to the “one-to-many” data access scenarios of a typical offline batch job——MapReduce applications. By analyzing the hardware parameters and historical load of each node in a heterogeneous cluster, the data blocks of the same erasure coded stripe are distributed as many as possible on nodes with similar performance. This way ensures that the data access pressure to each node of the cluster during the execution of the MapReduce job achieves relatively balanced state. In addition, when the system schedules tasks, the task concurrency of nodes is determined according to the current load and computing power of each node and so to avoid straggler task caused by heavy load in some nodes and optimize the progress of the MapReduce job. The experimental results show that compared with the default random data placement and task allocation strategy in Hadoop, the data layout strategy Heterogeneous-aware Data Placement Algorithm(HDPA) and the task allocation strategy Dynamic Task Allocation Algorithm(DTAA) proposed in this paper can effectively reduce the long tail effect of tasks in different types of MapReduce applications, thus reducing the running time by 10.5%~42%.