栅格地理数据并行格式转换引擎

Parallel geo-raster data conversion engine

如今大规模地理数据正在社会各个部门和组织中迅速积累,但是由于部门利益和历史沿袭等原因,大规模地理数据共享仍然极具挑战,相应共享技术需求仍然极其旺盛。作为地理数据共享的基础方式之一,传统单机地理数据格式转换技术,一方面受限于磁盘读写及带宽瓶颈,另一方面面对日趋庞大的数据规模,已很难满足实际应用需求。因此提出一种针对栅格地理数据的并行格式转换引擎,采用高性能计算集群环境支持大规模栅格地理数据转换共享,大幅降低了大规模栅格地理数据转换过程的时间成本。栅格地理数据并行格式转换引擎采用基于公共接口的设计理念,框架灵活、具有良好的扩展性,支持地理数据格式的读写自定义以及新数据格式添加,能够实现接入数据格式间的任意两两转换。为验证引擎框架及其处理效率,在Lustre并行集群环境下以格网数据交换格式(国家地理空间数据交换格式)向常见栅格地理格式的转换为示例进行了测试实验。结果表明,栅格地理数据并行格式转换引擎能够在8个节点Lustre集群中达到7.54的良好并行加速比。

Large scale geo-raster data have been accumulated all over the world in different departments and organizations during the past decades, but quite often in a variety of data formats, resulting in geospatial data sharing as an everlasting headache. Despite of various methodologies created, geospatial data conversion has always been a fundamental and efficient way for geospatial data sharing. However, as the size of data tends to be larger and larger, the methodology which was bounded by limited disk data transfer rate and bandwidth, needs a re-write and upgrade. A parallel geo-raster data conversion engine （PGRCE） was proposed to deal with massive geo-raster data sharing efficiently by utilizing high performance computing technologies. PGRCE was designed in an extendable and flexible framework, and was capable of customizing the way of reading and writing of particular spatial data formats. An experiment, in which geo-raster data in the CNSDTF-DEM format （ Raster spatial data defined in Chinese Geospatial Data Transfer Format Standard） were transferred using PGRCE in a parallel file system （ Lustre）, were conducted to validate the engine framework and its performance. Results show that PGRCE can achieve a 7.54 speedup on a Luster cluster of 8 nodes..