大数据存储技术在张衡一号卫星数据服务中的应用

Application of Big Data Storage Technology in Zhangheng-1 Satellite Data Service

随着张衡一号卫星监测数据的不断增加,目前的HDF5文件存储方式不仅在性能扩展性、读写并发性等方面凸显出劣势,而且无法实现对业务数据的精准时空查询,给用户带来很大困扰。为解决对海量卫星数据的高效存储和读取,分析了HBase数据库和ElasticSearch搜索引擎的优缺点,提出了一种基于二者相结合的卫星大数据存储方案,并搭建了Hadoop架构的大数据试验平台,完成了对张衡一号卫星空间电场超低频(ultra-low frequency, ULF)频段数据的存储试验和对比测试。结果表明,该方案大大提高了对海量卫星观测数据的读写并发性能,达到当前文件存储方式的几十倍,并实现了对卫星数据的精准定位和快速查询,很好地满足了用户要求。

With the continuous increase of the monitoring data of Zhangheng-1 satellite,the current HDF5 file storage mode not only highlights its disadvantages in terms of performance scalability,read-write concurrency and other aspects,but also fails to achieve accurate spatial-temporal query of the required data,which brings great difficulties to users.In order to solve the problem of efficient storage and reading of massive satellite data,the advantages and disadvantages of HBase and ElasticSearch engine were analyzed,a satellite big data storage scheme combining the two was proposed,and a big data test platform based on Hadoop architecture was built,and the storage test and comparative test of the ultra-low frequency(ULF)frequency band data of the space electric field of Zhangheng-1 satellite were completed.The results show that the scheme greatly improves the read and write concurrent performance of massive satellite observation data,which is dozens of times higher than the current file storage method,and realizes accurate positioning and fast query of satellite data,which well meets user requirements.