基于文件的大规模星表高效索引与融合

The Efficient Indexing and Fusion Algorithms for Large-scale Catalogs Based on File

大规模星表的快速检索是交叉证认、多波段数据融合、暂现源搜寻等任务实现的基础,尤其是大视场暂现源搜寻需要在一个曝光周期内完成观测结果与大规模星表的检索与交叉证认,以发现正在变化的天体。现有的大规模星表通常包含数十亿天体,为了在有限内存的情况下对其进行快速检索,提出了一套解决方案。通过使用基于HEALPix的多分辨率动态划分算法,能够将星表按照不同天区天体密度切分成大小合适且均匀的星表文件;进而在开源序列化组件Protocol Buffers基础上设计出一套针对星表的序列化方案,作为星表切分和检索时的中间存储介质,以尽可能提高检索时的速度。还尝试应用Peano-Hilbert编号代替HEALPix原有Z型编号顺序遍历星表,提高了缓存命中率,实现了对大规模星表的高效融合,方便对数据的后续利用与研究。

Transient source searching,which aims at discovering changing objects in the sky,requires wide-field telescopes to survey the sky continuously.After sources are extracted on images,they will be cross-matched with existing large-scale catalogs to detect which object is changing.This step must be very fast,or it will slow down the data processing and cannot retrieve real-time discovery.But current referenced catalogs,e.g.SDSS,Gaia,Pan-STARRS,contain billions of objects.It is very difficult to complete the cross-matching step in seconds using traditional methods.In this paper,we propose a solution for the fast retrieval of hundreds of GB or even TB of catalogs with limited memory.Catalogs will be indexed in forms of individual files instead of database.A multi-resolution dynamic splitting algorithm based on HEALPix is introduced.It divides the catalogs into appropriate and uniform files according to the density of objects in different sky regions.A searching scheme is also designed with this algorithm.In order to improve the file reading speed,we create a medium storage mechanism to save file.The mechanism is based on Protocol Buffers,an open source component.The Peano-Hilbert curve is also applied to replace the HEALPix’s original Zcurve,to fast traverse the catalog.With test,it effectively improves the cache hit ratio and data fusion efficiency on large-scale catalogs.With these improvements,billions objects searching and cross-matching in a limited hardware becomes possible.Our solution will help the implementation of changing object real-time detection,and other rapid detecting projects.