一种分布式异构带宽环境下的高效数据分区方法

An Efficient Data Partitioning Method in Distributed Heterogeneous Bandwidth Environment

在分布式大数据处理框架的作业运行过程中,会有大量的数据通过网络传输,数据在各节点之间传输所需的时间已成为作业运行的主要开销之一.在节点异构带宽的情况下,因为带宽瓶颈节点的存在,传统的数据分区方法效率低下.针对这个问题,建立了节点间的数据传输模型,该模型以降低数据传输时间为目标,根据各节点的上下行带宽和初始数据量大小,计算出各节点的最优数据分发比例.以该模型为基础,设计了基于带宽的数据分区方法,该数据分区方法使得各节点按最优数据分发比例来分配数据.最后在Apache Flink框架中将基于带宽的数据分区方法进行了实现,并通过实验进行了验证.实验结果表明:异构带宽条件下,基于带宽的数据分区方法可以有效减少数据分区所需的时间.

A large quantity of data is transmitted through the network during the process in distributed big data processing framework,resulting in the time consumption for data transmission between each node becomes one of the main costs of the operation.However,in the case of heterogeneous bandwidth of nodes,traditional data partitioning methods such as Hash partitioning or range partitioning will be inefficient,due to the existence of bandwidth bottleneck nodes.Data partitioning is necessary for big data processing and inefficient data partitioning methods would significantly increase the running time of jobs.We therefore propose a data transmission model between nodes to reduce time consumption in distributed heterogeneous bandwidth networks.The model calculates each node s optimal data distribution ratio to minimize the data transfer time,according to its uplink and downlink bandwidth as well as the initial data size.Besides,a bandwidth-based data partitioning method is designed based on the proposed model,enabling each node to allocate data under the optimal data distribution ratio.We demonstrate the effectiveness of our bandwidth-based data partitioning method through the implementation in the Apache Flink framework and have significantly improved efficiency.Extensive experimental results show that the bandwidth-based data partitioning method can effectively reduce the time consumption of data partitioning in distributed heterogeneous bandwidth conditions.