利用MPI构建柔性数据处理系统

USING MPI TO CREATE FLEXIBLE DATA PROCESSING SYSTEM

为了提升在线应用程序进行大数据量统计时的速度,设计独立的基于消息传递接口MPI(Message Passing Interface)的并行数据处理服务模型,与客户应用程序一同建立了柔性数据处理系统。阐述客户程序和并行服务系统的互联结构和工作方式,按照"先判断后处理"的工作思想,客户程序根据运算规模确定是否调用并行服务程序。并行服务程序由监听程序、临时工作程序和一组用于服务的并行程序组成,设计了基于可扩展标记语言的通信协议,实现了并行服务系统与客户程序之间的数据交换。实验比较了串行、MPI并行和Java多线程程序的耗时,结果表明MPI并行程序比其他两种方式具有明显的优势。

To accelerate the statistics of large amount of data processed by online applications, the paper designs a parallel data processing service model which is based on the Message Passing Interface （MPI） technology. The model, together with the client application, forms a flexible data processing system. The paper elucidates the connection structure between the client applications and the parallel service system as well as the operation mode. The working principle is that an operation is executed after the scale of the problem is clear. The client appli- cation invokes the parallel service program only when the problem scale is large enough. The parallel framework consists of a listener, a tem- porary worker and a group of parallel service programs. Communication protocols based on Extensible Markup Language are designed for data exchange between the parallel servicesystem and the client application. Experiments are carried out to compare the sequential program, the MPI program and the Java multi-thread program in their time consumption. Results show that the MPI parallel program is apparently faster than the other two approaches.