IoT多源异构大数据存储与管理设计

IoT Multi-source Heterogeneous Big Data Storage and Management Design

目前,物联网数据呈现规模大、来源多、结构不一、冗余度高等问题,导致数据读写、存储和管理困难。通过分析在物联网多源异构数据的存储和管理上,本文总结出三个关键问题:数据完整性问题、负载倾斜问题和数据压缩问题,并提炼出对存储侧的性能要求。针对以上问题,对检索(ElasticSearch)、存储(HBase)、数据压缩等技术进行了分析,并提出设计方案。在设计方案中,提出数据存储前需要进行分类,丢弃不可用数据、压缩冗余数据、将常规数据进行冷热分区;在数据存储时设计Rowkey、优化索引;在数据存储后进行异常存储检测。通过正面物联网多源异构大数据存储和管理技术面临的窘境,提出设计建议,为今后的智慧交通、智慧小区、智慧城市等场景落地提供有力的借鉴。

At present,Internet of Things data presents problems such as large scale,multiple sources,different structure and high redundancy,which leads to difficulties in data reading and writing,storage and management.Through analysis,in the storage and management of multi-source heterogeneous data in the Internet of Things,this paper summarizes three key problems:Data integrity,load tilt and data compression,and extracts the performance requirements on the storage side.In view of the above problems,ElasticSearch,HBase,data compression and other technologies are analyzed,and a design scheme is proposed.In the design scheme,it is proposed that the data should be classified before storage,discard the unavailable data,compress the redundant data,and divide the conventional data into cold and hot zones;Designing Rowkey and optimizing index in data storage;Abnormal storage detection is performed after data storage.Through the dilemma faced by the multi-source heterogeneous big data storage and management technology of the Internet of Things,the design suggestions are put forward to provide a powerful reference for the future intelligent transportation,smart community,smart city and other scenes.