基于Flink+QLExpress处理海量数据的实时监控告警系统

Real-time Monitoring and Warning System Based on Flink and QLExpress to Process Massive Data

为了满足大数据监控告警系统具备灵活告警及实时动态更新各类配置的需求,在深入研究各大数据监控告警系统架构的基础上,提出一种基于Flink+QLExpress新架构的处理海量数据的实时监控告警系统。首先通过Canal采集数据,然后利用聚合程序聚合数据后推送至Kafka,利用Flink从Kafka数据源获取源数据,并通过Flink SQL技术实现数据清洗及过滤,得到满足阈值条件的初级告警数据,最后通过QLExpress技术实现告警数据的二级过滤、告警信息的格式化及推送。结果表明此架构通过引入Flink SQL和QLExpress能够实现告警口径、告警阈值、告警格式及推送方式的动态更新与发布,增强了告警的实时性、灵活性及便利性,提升了运维效率。

In order to meet the requirements of flexible warning and real-time dynamic updating configuration of massive data monitoring and warning system,a real-time monitoring and warning system based on the new architecture of Flink+QLExpress is proposed based on the in-depth study of the architecture of various massive data monitoring and warning systems.Firstly,data is collected by Canal,then aggregates by the aggregator program and pushed to Kafka.Flink takes Kafka as the data source to obtain the origin data,and realizes data cleaning and filtering through Flink SQL technology to obtain the primary alarm data that meets the threshold conditions.Finally,QLExpress technology is used to realize the secondary filtering of alarm data,the formatting and pushing of alarm information.The results show that by introducing Flink SQL and QLExpress,this architecture could realize the dynamic update and release of alarm caliber,alarm threshold,alarm format and push mode,enhance the real-time,flexibility and convenience of alarm,and improve the efficiency of operation and maintenance.