基于Spark的大规模天文数据天区覆盖生成算法

A Sky Coverage Generation Algorithm for Large-scale Astronomical Data Based on Spark

天区覆盖生成是天文数据归档中的重要一环,其结果对天文数据检索、计算等后续处理流程至关重要.由于天文数据的海量性,应用传统科学计算方法处理这一问题通常耗时较长,效率不高,且受存储空间的制约,扩展性差.为解决这一问题,本文提出了一种基于HEALPix索引和Spark框架的高效分布式天区覆盖生成算法.实验证明:该算法可以在短时间内完成大规模天文数据的天区覆盖生成,为实现海量天文数据的快速归档提供了支持;同时,所生成的结果还可以用于数据可视化,直观地展现星表中的天文数据在天区上的分布情况.

The sky coverage generation is an important part of astronomical data archiving.Its results are very important for the subsequent processing of astronomical data retrieval and calculation.Due to the massive nature of astronomical data,it is usually time-consuming and inefficient to deal with this problem with traditional scientific computing methods,and it is limited by the constraints of storage space and poor scalability.In order to solve this problem,this paper presents an efficient sky coverage generation algorithm based on HEALPix index and Spark.Experiments show that the algorithm can complete the generation of large-scale astronomical data in a short time.The generated results of the algorithm can be used to accelerate the processes of dealing with astronomy data and can also be used for data visualization of the sky coverage.