面向大数据应用的多层次混合式并行方法

Multilevel hybrid parallel method for big data applications

基于很多大数据应用存在对数据进行多种并行处理的需求,提出两层混合式并行方法,即执行单元的混合并行和计算模型的混合并行.通过在同一个计算节点上执行单元的混合并行,充分挖掘基础设施的计算能力,从而提高数据处理性能;采用在同一个执行引擎中集成多个计算模型的并行方法,以适合应用多样异质处理模式.不同的混合并行方法可以契合不同的数据和计算特点,以满足不同的并行目标.介绍了混合式并行方法的基本思想,并以前期开发的并行编程模型BSPCloud为基础,阐述了进程和线程混合并行、BSP和Map Reduce混合并行的主要实现机制.

Many large data applications require a variety of parallel data processing. This paper presents a two-layer hybrid parallel method, i.e., hybrid parallel of execution units and hybrid parallel of computing model. By hybrid parallel of execution units on the same computing node. The computing power of infrastructure can be fully taped, and thus data processing performance can be improved. By integrating several calculation models into the same execution engine in a parallel way, diverse heterogeneous processing modes may be applied. Different hybrid parallel ways can meet different data and calculation characteristics, and meet different parallel objectives as well. This paper introduces the basic ideas of hybrid parallel methods, and describes main implementation mechanisms ofhybrid parallelism.