省级气象大数据存储模型设计

Design of provincial meteorological big data storage model

从结构化、非结构化气象数据的存储、索引两方面,基于大数据相关技术设计一套省级气象大数据存储模型。结果表明,非结构化气象小文件利用Avro进行数据模式设计、依据存储规则合并打包为Avro格式文件存储在HDFS上,减少文件数,节省内存资源的消耗;规则的结构化气象数据利用HBase提出表、列字段、行键设计的宽表存储设计思路;利用ElasticSearch设计统一的省级气象大数据存储索引库,统一检索入口,提高检索效率。

Referring to the mature meteorological data storage ideas of traditional storage technology and the research results of meteo⁃rological big data storage technology of meteorological departments at all levels at present,a set of provincial meteorological big data storage model based on big data related technology from the storage and index of structured and unstructured meteorological data was designed:the results showed that unstructured meteorological small files use Avro to design data mode,According to the storage rules,are merged and packaged as the Avro format files and stored on HDFS to reduce the number of files and save the consumption of memory resources;Based on HBase,put forward the design idea of wide table storage for regular structured meteorological data,including ta⁃bles,column fields and row keys;Elasticsearch is used to design a unified provincial meteorological big data storage index library to unify the retrieval entrance and improve the retrieval efficiency.The design ideas and the use of related technologies in this paper can provide reference for provincial meteorological departments to carry out the storage planning and scheme formulation of meteorological big data.