摘要
随着天文设备的进步发展,对海量天文数据处理形成了新的挑战。为使当前主流的天文数据分析软件可以有效处理海量数据,提出一种适用于射电天文处理上层应用的分布式数据存储策略DDSS(Distributed Data Storage Strategy)。首先,设计分布式数据存储策略的系统框架。其次,设计混合分片列式存储方法,在保留列存储查询优势的同时提升了数据导入的速度。进一步,通过维护基于相对位置映射的元数据来快速读取包含大量数据的天文阵列数据,显著地提升了天文处理应用底层数据读写的吞吐量。最后,通过实验证明了该方法的有效性。
With the progress and development in astronomical equipments,to process massive astronomical data forms the new challenge. In order to make current mainstream astronomical data analysis software can effectively deal with massive data,we proposed a distributed data storage strategy( DDSS) applicable for radio astronomy to deal with upper application. First we designed the system framework of DDSS. Then we designed the hybrid method of partitioning and columnar storage,while preserving the advantage of column-store in data query,the speed of data import is promoted as well. Furthermore,by maintaining the relative position mapping-based metadata to quickly read astronomical array data containing large amount of data,the I / O throughput of astronomy when processing underlying data of applications is significantly improved. Finally,through experiment we testified the effectiveness of the proposed method.
出处
《计算机应用与软件》
CSCD
2016年第11期28-32,共5页
Computer Applications and Software
关键词
分布式
天文
阵列数据
Distribution
Astronomy
Array data
