Spark任务间消息传递方法研究

Exploring Message Passing Method Between Spark Tasks

当今诸多工程问题及科学研究中,都面临着大数据处理和高性能计算任务的双重挑战。基于内存计算技术提出的分布式处理框架Spark已在学术和工业界得到了广泛的应用,但其MapReduce-like的编程模型在任务间无法进行通信,导致科学计算中的数值算法无法进行高效实现。针对上述问题,研究了一种Spark内存计算与MPI消息传递模型相结合的解决方案,充分利用内存访问存取快速的特点和MPI的多种高性能通信机制,解决了Spark编程模型表达能力不足的缺陷,同时为MPI提供了面向数据的DAG计算方式。通过对Spark内部的运行环境和调度系统进行修改,使得MPI在Spark中得以无缝融合,为高性能计算和大数据任务提供了一个统一的内存计算系统。测试结果表明,在数值计算和迭代算法上相比Spark至少有50%的性能提升。

Engineering problems and scientific research are facing dual challenges of big data processing and highperformance computing tasks.Spark,a distributed processing framework based on in-memory computing technology,has been widely used in academia and industry.However,its MapReduce-like programming model fails to communicate between tasks,causing numerical algorithms in scientific computing cannot be efficiently implemented.In response to the above problems,a computing system is proposed in this paper that combines Spark in-memory computing model with MPI message passing,which takes full advantage of the fast speed of memory access and multiple high performance communication mechanisms of MPI.It can not only supplement the insufficient expressiveness of the Spark programming model,but also provide a data-oriented DAG computation method for MPI.Internal runtime environment and scheduling strategy of Spark are modified to seamlessly integrate MPI into Spark to provide a unified in-memory computing system for high-performance computing and big data processing tasks.The tests indicate that the performance of numerical computation and iterative algorithm is improved by at least 50%compared with Spark.