用MapReduce框架构建虚拟天文台数据节点

Constructing Data Nodes of the China-VO with the MapReduce

MapReduce是一种大规模分布式并行处理框架,最初被用于互联网服务中的海量数据处理,并逐渐扩展到各个行业领域。目前,虚拟天文台面临着越来越多的地面及空间望远镜观测到的海量天文数据。为了提高中国虚拟天文台数据节点处理海量天文数据的能力,首次提出基于MapReduce框架构建中国虚拟天文台数据节点的方法,并以批量星表交叉认证为例描述了具体实现过程,性能评估结果证明基于MapReduce框架构建虚拟天文台数据节点,可以在性能、扩展性与成本等多方面获得收益。

The MapReduce is a distributed parallel processing model and execution environment for processing large data sets. It was initially applied to handle massive data in web service, but its applications have been extended to a variety of areas. A current project of Virtual Observatory may face an increasingly massive amount of astronomical data from ground-based and space telescopes. In order to improve the processing capacity of the astronomical data center in the China Virtual Observatory, this paper proposes a new approach to construct data nodes using the MapReduce. It translates an astronomical query to a standard SQL query, and then turns the query into a MapReduce job. It finally outputs the results in the standard formats of astronomical data. The MapReduce is integrated into the China Virtual Observatory by using the above three steps. Because cross-identifying between object catalogs takes place only once, the main consumed time in the MapReduce is in indexing and calculating data. We implement object cross-identification based on the MapReduce framework and our performance evaluation shows that the MapReduce-based cross-identification outperforms the traditional approach based on DBMS. Our results also show that the MapReduee-based framework achieves not only good performance but also scalability and low cost.