基于Parallel NetCDF的海量时空数据存储研究

Storing Massive Spatio-temporal Data Using Parallel NetCDF

随着遥感、无线传感器网络等技术的发展,海量时空数据存储的需求越来越大,如何将长时间序列时空数据在集群上进行并行存储和处理已成为科研工作者要解决的课题。本研究深入分析了时空数据特点及NetCDF文件结构特点,利用Parallel NetCDF函数库,基于集群环境将GRID数据并行转换为NetCDF数据。以黑河干流流域月平均温度GRID数据为例验证并行转换算法的有效性,并可以扩展到其他时空数据的并行存储。

With the technical development of remote sensing, wireless sensor network, ect., the needs for massive spatio-temporal data storage are increasing dramatically.It has become a big problem for the researchers that how to store and process the long time sequence data on a computing cluster. We analyzed the spatio- temporal data characteristics and NetCDF file structure features, and using parallel NetCDF function library, we implemented the conversion of GRID parallel data to NetCDF data based on the clusterenvironment. Taking monthly mean temperature GRID data of Heihe basin as an example, it was shown that the parallel conversion algorithm is effective, and can be extended to the parallel storage of other spatio-temporal data.