智能配用电大数据存储架构设计

Design of storage framework for big data of SPDU

针对智能配用电数据海量、高维度、多源异构等特点,提出一种大数据存储的三层管理框架设计方案。首先,对智能配用电数据组成进行分类划分。然后,提出智能配用电数据存储的资源层、元数据层和实际数据层的三层管理框架,在资源管理层,应用虚拟化技术、负载均衡和资源调度完成对物理资源的管理;在元数据管理层,使用数据预处理技术对智能配用电的元数据进行分类,采用中间件技术完成XML元数据仓库管理;在实际数据管理层,利用NoSQL技术,通过Map和Reduce的有效结合,完成海量数据的分层分区快速存储。最后,在实验室环境下对该设计方案进行初步的应用,验证了所设计方案能够对结构化、半结构化和非结构化数据进行统一存储,可为智能配用电大数据技术的应用提供基础支撑作用。

A three-layer management framework is designed for the storage of massive,highly dimensional and heterogeneous data of SPDU（Smart Power Distribution and Utilization）. The data components of SPDU are classified and a hierarchical management framework with resource,metadata and actual-data layers is proposed for the storage of SPDU data. The virtualization technology,load balancing and resource scheduling are applied in the resource management layer to realize the management of physical resources. The data preprocessing and middleware technologies are applied in the metadata management layer to respectively classify the metadata of SPDU and manage the warehouse of XML metadata. The NoSQL technology is applied in the actual data management layer to fast store the massive data into different layers and divisions through the effective combination of Map and Reduce. The proposed design scheme is preliminarily applied in the laboratory environment to verify its unified storage of structuralized,semi-structuralized and non-structuralized data,which provides a basic support for the application of big data technolozv to SPDU.