基于Flink的空间科学卫星数据实时处理方法

Real-Time Processing Method of Space Science Satellite Downlink Data Based on Streaming Computing

为满足地面数据处理系统高吞吐率、低延迟地处理空间科学卫星下行数据的需求,提出一种基于流式计算的空间科学卫星下行数据实时处理方法。方法借助消息中间件Kafka的分区实现数据分流,分流后的数据依次流入Flink集群中的并行单元实现并行处理。使用GECAM空间科学卫星下行数据进行仿真,结果表明,上述方法具有较高的吞吐率和较低的延迟,并且可以通过扩展服务器数量实现更高吞吐率和更低延迟,满足空间科学卫星下行数据的高速实时处理需求。

Aiming to meet the needs of ground data processing system to process the downlink data of space science satellites with high throughput and low delay,a real-time processing method of downlink data of space science satellites based on stream computing is proposed.The message middleware Kafka is used to achieve data shunting.The distributed data flows into the parallel units in the Flink cluster in turn to realize parallel processing.The simulation experiment data was based on the real downlink data of the GECAM space science satellite during the scientific missions.The simulation result shows that this method has high throughput and low delay,can achieve higher throughput and lower delay by expanding the number of servers,and meets the demand for high-speed real-time processing of downlink data from space science satellites.