基于Spark Streaming的明安图射电频谱日像仪实时数据处理

Real-Time Data Processing in Mingantu Ultrawide Spectral Radio Heliograph Based on Spark Streaming

目前天文观测中对数据的实时处理需求越来越多,性能要求也越来越高,我国明安图射电频谱日像仪(Mingant U Sp Ectral Radioheliograph,MUSER)是同时以高时间、高空间和高频率分辨率对太阳进行射电频谱成像的设备。在低频部分的日常观测中,包含了两方面的需求:(1)对历史数据的处理;(2)5秒钟抽样观测数据的处理。抽样观测数据需要实时处理,并在监控终端显示,数据处理过程包含了数据校验、修正、成图、洁化等多个步骤,传统的单机处理模式已无法满足大数据量下的实时性要求。因此,实时数据计算中,使用Spark Streaming流式计算这一新兴的分布式计算方法,设计了自定义的接收器,并将多个图形处理器节点加入到分布式集群中。通过实验对性能进行评估,结果证明基于内存的高速执行引擎的特点能显著提高性能。期待能通过实验进一步优化算法和配置,获得更好的结果,并最终运用到实际环境中。

There is a growing demand for real-time processing in astronomical observations in recent years, meanwhile, the requirement for performance is also increasing dramatically. Mingantu Ultrawide Spectral Radio Heliograph （MUSER） is a synthetic aperture radio interferometer with high temporal, spatial and spectral resolution. In daily observation of low frequency, MUSER contains two aspects of data processing, historical data processing and sampling observational data which is produced every 5 seconds and processed in real-time mode. The procedure of raw data processing contains validation, correction, clean and other processing steps, then the results need to be transmitted in real-time mode to monitoring end without user constantly refreshing or sending a request. The traditional stand-alone processing mode has been unable to meet the requirements of large amounts of data in real-time mode. In this paper, we explored the use of Spark Streaming in a new approach for MUSER real-time calculations across multiple machines and evaluated its effectiveness and efficiency. A customized receiver was created for real-time binary stream of MUSER. We also extended the Spark cluster by adding multiple GPU＇s nodes. The experiments have shown that Spark Streaming can significantly improve MUSER real-time processing performance for its memory-based execution engine. We might look forward to optimize the algorithm through experiments and configurations so as to obtain better results, and apply it to the actual environment of MUSER finally.