基于HBase的后勤保障监控数据实时检索方案设计

Design of real-time retrieval scheme for logistics support monitoring data based on HBase

后勤综合保障监控数据有着实时性、流速快、海量以及多维度的特征,对于监控数据的多维度检索、分析和预警都有较高的实时性要求。鉴于此,文中基于HBase设计了一种分布式监控数据实时存取系统。通过Kafka Streams进行流数据清洗解码,并利用ElasticSearch构建二级索引优化查询。实验表明该系统对PB级数据多维度检索性能提高10~30倍,方案可行且高效。

The characteristics of logistic support monitoring data are real-time,fast flow rate,mas-sive and multi-dimensional.And there are high real-time requirements for multi-dimensional retrieval,analysis and early warning of monitoring data.Therefore,a distributed monitoring data implementation access system based on HBase is designed.Kafka Streams is used for stream data cleaning and decoding,and a secondary index optimization query is built by ElasticSearch.Experiment results show that the system improves the multi-dimensional retrieval performance of PB data by 10~30 times,so the scheme is feasible and efficient.