并行算法在气象数据处理中的研究与应用

Research and application of the parallel algorithm in meteorological data processing

针对目前单进程和半并行的气象数据加工处理方式存在时效差、延迟高的问题,设计了一套全并行算法,该算法利用ASCII码将数据文件名转换为一个固定值,通过哈希函数中的除留余数法将数据分配给任务数最少、处理速度最快的子节点。为了加快数据的处理速度,将每一个大数据包切分为多个小数据包,将小数据包调度到不同的节点进行处理,待数据加工处理完之后以消息的形式告知消费者。测试结果表明,在相同的硬件资源条件下,使用全并行处理方式比单进程处理方式的效率提升了约9倍,比半并行处理方式的效率提升了约1.54倍,在实际业务应用中取得了良好的效益。

Aiming at the problems of poor timeliness and high delay in the current single⁃process and semi⁃parallel meteorological data processing methods,a set of fully parallel algorithms is designed.In this algorithm,the data file name is converted into a fixed value by ASCII code,and the data is allocated to the child node with the least number of tasks and the fastest processing speed by the division and remainder method in the hash function.In order to speed up data processing,each large data packet is divided into multiple small data packets,and the small data packets are scheduled to different nodes for processing.After data processing,consumers are informed in the form of messages.The test results show that under the same hardware resources,the efficiency of fully parallel processing mode is about 9 times higher than that of single process processing mode and 1.54 times higher than that of semi⁃parallel processing mode.Good application benefits have been obtained in practical business applications.