大数据平台实时计算监控方案分析

Analysis of Real-time Computing Monitoring Scheme for Big Data Platform

在实时业务数据处理中,需要用到Flume、Kafka、Slipstream等一系列功能组件。在前台业务系统与大数据平台的数据传输过程中,需要监控数据是否正常流动。如果出现了异常,需要定位异常出现的位置。通过一系列的监控工具和方法,可以快速定位各个环节的功能组件是否正常运行。该技术方案的主要目的是为了监控大数据平台实时数据计算的整体流程是否正常工作,前台业务系统与大数据平台内部的数据校验、大数据平台内部的数据校验逻辑是否正确。一方面可以在日常开发过程中验证数据的可靠性,另一方面也可以帮助运维人员更快更精准地定位到产生异常数据的问题所在,从而对现有的业务逻辑进行优化,提高业务效率。

In real-time business data processing,we need to use flume,Kafka,slipstream and other functional components. In the process of data transmission between foreground business system and big data platform,we need to monitor whether the data flows normally. If there is an exception,we need to locate the location of the exception.Through a series of monitoring tools and methods,we can quickly locate whether the functional components of each link operate normally. The main purpose of this technical scheme is to monitor whether the overall process of realtime data calculation of big data platform works normally,whether the data verification logic of front business system and big data platform is correct,and whether the data verification logic of big data platform is correct. On the one hand,it can verify the reliability of data in the daily development process,on the other hand,it can help the operation and maintenance personnel locate the problem of abnormal data faster and more accurately,so as to optimize the existing business logic and improve the business efficiency.