基于开源框架的电力大数据云端实时数据采集平台的设计与关键技术

Design and Key Technology of Power Big Data Cloud Real-Time Data Acquisition Platform Based on Open Source Framework

笔者在分析大数据相关开源框架和技术的基础上,提出基于开源框架实现边缘侧数据汇聚接入大数据云端、云端数据采集、分布式流处理、实时数据计算以及数据存储的框架设计与关键技术应用。该功能设计在某大型能源企业电力大数据系统中已成功实践应用,有效解决了实时数据采集应用中数据量大、响应速度快、数据传输可靠性要求高等问题。

Based on the analysis of open source frameworks and technologies related to large data,this paper proposes the framework design and key technology application of edge-side data aggregation and access to large data cloud,cloud data acquisition,distributed stream processing,real-time data calculation and data storage based on open source frameworks.The function design has been successfully applied in a large power data system of a large energy enterprise.It effectively solves the problems of large amount of data,fast response speed and high reliability of data transmission in the application of real-time data acquisition.