云计算平台上实现30年气候资料整编的方法

Method for Thirty-years Climate Normals Generation Based on Cloud-computation Platform

国家气象信息中心存储和保存了50多年宝贵的长序列历史资料,这些历史资料在实时、准实时业务及科研中需要经常被使用并进行气象科学计算。由于历史数据量大,耗时长,如何在短时间内得到所需的计算结果提供用户使用成为本文的主要研究目标。通过搭建云计算平台,并以30年气候资料统计整编研究对象,在云计算平台上基于MapReduce分布式并行计算模型进行多种统计项目、统计方法的算法实现。通过修改云计算平台运行环境参数配置并在不同配置下运行相同计算任务,进行计算效率对比试验。

Cloud computation technologies, which has solved the problem of low computation power of a stand alone server, uses distributed computation technology to achieve the computation power of parallel computation and computational efficiency. National Meteorological Information Centre has stored and saved more than fifty years of valuable and precious long sequence of historical data, which often be used in real-time, near-real-time business and research needs and meteorological scientific computation. The amount of historical data is so large and the time of consuming is so long that it has become the main target of this paper to solve how to get the required calculation results in a short or limited time. Based on Hadoop cloud computation platform this paper has built a cluster mode and take the thirty years of climate statistics reorganization for example, realizes a variety of statistical methods and statistical method algorithm using MapReduce computation model. This paper also modifies the Hadoop cluster environment configurations, by starting a different number of task nodes to record different computational efficiency comparison data.