基于大数据的航道采集系统的架构设计

Design of Waterway Collection System Based on Big Data

随着大数据技术的迅速发展,智能航道应运而生。航道的水文、水深、岸线等要素数据是实时监测获取的,其具有大数据4V的特征。同时,航道行业中没有标准统一的通讯协议,智能采集设备和业务系统高度耦合,各业务系统存在信息孤岛的状况,无法满足大数据量的要求。因此,本文提出一种基于大数据的航道采集系统的架构设计,构建了高通量、高可靠、高保密的航道数据的采集通道。该系统架构包含四个中心模块:采集处理中心、指令下发中心、数据生命周期管理中心和配置管理中心。各中心模块采用大数据技术分布式/集群架构,能够高效、可靠的处理大规模的实时数据,且能根据数据的规模动态调整集群的大小。本系统通过配置管理中心动态配置协议、设备,实现了对智能设备的可插拔式的管理。所有的智能设备监测的数据均通过采集系统处理后,由业务系统从分发模块订阅,降低设备与业务系统耦合度,且实现所有数据的汇聚整合,有利于后期航道大数据的分析挖掘,进而发现更多有重要价值的航道知识。

With the rapid development of big data technology, intelligent waterway came into being. The hydrology, water depth, shoreline and other elements of the waterway are being monitored and acquired in real time, which have the 4V characteristics of big data. At the same time, there is no standard unified communication protocol in the waterway industry, intelligent collection equipment and business systems are highly coupled, and the information island situation exists in each business system. The above situations can＇t meet the requirements of big data. Therefore, the design of wate rway collection system based on big data was proposed, in order to build the high-throughput, high reliability and high confidential waterway collection channel. The architecture consists of four modules： collection and processing module, order delivery module, lifecycle management module and configuration management module. Each module uses distributed / cluster architecture of big data technology, which can efficiently and reliably handle the large-scale and real-time data, and can dynamically adjust the size of the cluster according to the scale of the data. The configuration management module can be used to configure the protocol and equipment, to realize the pluggable management of intelligent devices. All monitoring data are processed by the collection system, and then the business system subscribes from the distribution module, in order to reduce the coupling degree and realize the integration of all the data, which is helpful for the analysis and mining of the waterway data, and discover more valuable waterway knowledge.