多源异构遥感大数据的高性能存储技术研究

Research on Storage Performance Improvement Technology of Multi-source Heterogeneous Remote Sensing Big Data

为了提高多源异构海量遥感数据处理流程中存储系统的性能,研究了自适应内存缓存机制在遥感数据存储中的应用.首先使用Alluxio建立统一的虚拟文件系统,并研究了基于空间关系及历史数据访问推理的缓存机制.其次引入了基于内存运算的Presto作为遥感数据元信息的存储组件,为海量的遥感数据元信息及高级语义信息提供实时的读写支持.实验结果表明,相对于使用传统存储方式的流程,使用自适应内存缓存遥感数据存储系统的流程处理耗时减少18%以上,该存储系统对提高遥感数据处理效率贡献显著.

In order to improve the performance of storage system in multi-source heterogeneous massive remote sensing data processing,this paper studied the application of adaptive memory caching mechanism in remote sensing data storage.First,Alluxio is used to establish a unified virtual file system,and at the same time the caching mechanism based on spatial relationship and historical data access reasoning is researched.Secondly,the memory-based Presto is introduced as a storage component of remote sensing data meta-information,providing real-time reading and writing support for massive remote sensing data meta-information and high-level semantic information.Compared with the workflow processed by the traditional storage method,the experimental results show that the processing time of the workflow with data storage system based on adaptive memory caching is reduced by more than 18%,the storage system contributes significantly to the improvement of the processing efficiency of remote sensing data.