读写分离技术架构下海量数据的索引构建研究

Research onindex construction of massive data under the framework of read-write separation technology

针对电力系统安全生产管理平台由于数据量的增加导致系统性能下降的问题,研究了基于读写分离技术架构下的海量数据索引方法。在分析LSM-Tree索引技术的基础上,提出了面向读写分离、批量更新存储机制的LSM-Index分布式索引方法。在此基础上,利用静态索引和增量索引的方式构建了局部索引与全局索引框架。并基于map-reduce排序方式完成了大数据的静态索引建构,同时利用并行任务调度模型对其进行了优化,实现了数据的快速存取和检索。本研究内容对于提高电力系统安全生产管理平台数据处理能力,改善系统性能具有重要意义。

In this paper,the method of mass data indexing based on read write separation technology is studied in order to reduce the performance of the system due to the increase of data amount. Based on the analysis of LSM-Tree index technology,a distributed LSM-Index indexing method is proposed,which is oriented to read and write separation and batch update storage mechanism. On this basis,a local index and a global index framework are constructed by using static index and incremental index. The static index construction of large data is completed based on the map-reduce sorting method. At the same time,the parallel task scheduling model is used to optimize it,and the fast access and retrieval of data are realized. The research contents in this paper are of great significance for improving the data processing ability of power system safety production management platform and improving the system performance.