基于消息队列的多进程数据处理系统

Multi-process data processing system based on message queue

针对当前数据库中海量数据的处理问题,提出了多进程的处理方式,设计并实现了一个基于消息队列的多进程数据处理系统。介绍了该系统的原理,给出了系统的总体构架,并在Unix操作系统下实现了系统功能。该系统将消息队列作为进程间通信的手段,同时提供了对各进程状态进行监控的用户界面。采用的建造者设计模式在功能上具有较好的扩展性。通过实验结果表明,该系统相比串行的处理模式大幅减少了系统运行时间,为数据库中数据的处理提供了可行的解决途径。

Aimed at the problems of current mass data in database,a multi-process processing method is presented and a complex data processing system,which uses message queue as the inter process communication（IPC） tool is designed and implemented.The principle and architecture of the system is explained and the system is implemented under the unix operating system.The UI of monitoring every process status is provided and the builder design pattern is used,so the system has more extensibility.Finally,experimental results show that this method is better than traditional ones in reducing the system execution time and feasibility of the method is validated.