基于大数据技术的气象算法并行化研究

Research on Parallelism of Typical Meteorological Algorithm Based on Big Data Technology

在气象数值预报解释应用业务中,传统数值算法的应用呈现逐步增加的趋势,但是随着算法输入数据种类和数据量的增加导致算法的完成时间大幅增长,甚至出现了算法完成时间性能瓶颈。为了突破算法时间上的性能瓶颈,基于OpenCV算法库,实现了多元逐步回归和卡尔曼滤波算法的执行模块,采用Map-Reduce计算框架设计和实现了多站点输入数据分割的并行化执行模块;规范了算法输入和输出的数据格式,设计了并行算法的Web服务流程以及实现了基于Rest Web Service的算法访问接口。业务应用实验测试表明,并行算法能够很好地满足气象业务实际需求。

The traditional application of numerical algorithms is widely used in the interpretation application of numerical weather predic- tion. However with the increasing of the type and amount of algorithm input data, the completion time of it presents exponential growth, which faces the bottleneck in the algorithm completion time performance. In order to break it, an execution module of stepwise multiple regression and Kalman filter algorithm is developed based on the OpenCV library （Open Source Computer Vision Library）. A distributed data storage model and parallel data access service are designed based on Hadoop framework, and the parallel strategy is designed based on the Map-Reduce framework,then the parallelism execution module is achieved based on the implement of a multi-site input data par- tition. The algorithm input and output data format are standardized. A Rest Web Service interface of parallel algorithm is designed. Opera- tional trial in multi-user environment shows that this algorithm can meet the actual requirements of meteorological business greatly.