风云三号气象卫星大数据集产品并行处理模式

Parallel processing mode for big data set production of Fengyun-3 meteorological satellite

针对风云气象卫星地面应用系统中如何缩短大数据集产品的处理时延,提高系统资源利用率的问题,提出了一种基于性能预测的、计算节点作业执行概率的信息素动态加权并行算法(pdw),并构建了新的并行处理模式。该模式一方面能够根据轨道级遥感数据与目标产品之间的空间位置匹配关系进行网格处理粒度的合理划分;另一方面,采用pdw算法综合考虑节点本身的处理能力和实时负载信息进行分粒度作业的资源匹配调度。业务实测结果表明,较传统的处理模式,采用新的处理模式后,具有典型大数据集特征的风云三号(FY-3)中分辨率光谱成像仪(MERSI)250 m分辨率全球植被指数分层数据格式(HDF)产品的处理时效提高了10倍左右,所提并行处理模式可以有效缓解I/O资源占用瓶颈、均衡系统负载,提升大数据集产品的处理效率。

In view of the issue that how to reduce the latency of processing the big data set products and improve resource utilization of cluster system in the Fengyun meteorological satellite ground application system,a scheduling algorithm called pheromone with dynamic weight（ pdw） based on performance prediction and calculating the probability of the node being selected to run the jobs was proposed,a new parallel processing mode was also built based on the algorithm. On the one hand,in the parallel processing mode,the space grid processing granularity of remote sensing data was divided by the space matching relation between the orbit level data and target products. On the other hand,the data processing jobs were scheduled considering the processing power of the node itself and real-time load by the pdw algorithm. The operational results show that,the processing efficiency for Fengyun-3（ FY-3） Medium Resolution Spectral Imager（ MERSI） 250 m resolution MERSI Vegetation Index（ MVI） Hierarchical Data Format（ HDF） globe products is increased by about 10 times compared the traditional mode,the proposed parallel processing mode can effectively alleviate bottleneck of I / O resource,balance system load and improve the processing efficiency of big data set products.