基于协处理器的HBase内存索引机制的研究

Research and Implementation of HBase Memory Indexing Scheme Based on Coprocessor

为了实现对海量数据的高效存储和查询,众多NoSQL数据库被开发出来,HBase是其中之一。但原生的HBase数据库在进行数据查询时只支持主键索引,对非主键数据只能通过全表扫描的方式进行查询,极大降低了HBase的多条件查询速度。为此,提出了基于协处理器的HBase内存索引构建方案,通过协处理器实现对二级索引的快速构建并可根据HBase表的变化自动更新索引。同时,将建立的索引进行持久化操作,在使用时通过内存计算,极大地提高了索引数据检索速度,保证了索引的可用性和容错性。实验结果表明,该方案相比原生数据库的条件检索速度有了极大提升,相比于基于Solr和HiBase的二级索引方案检索速度也有所提升。

In order to achieve efficient storage and query of massive data,many NoSQL databases have been developed,and HBase is one of them.However,the native HBase database only supports the primary key index when performing data query,and the non-primary key data can only be queried by means of full table scan,which greatly reduces the multi-condition query speed of HBase.To this end,a HBase memory index construction scheme based on coprocessor is proposed.The coprocessor is used to quickly construct the secondary index and the index can be automatically updated according to the change of the HBase table.At the same time,the established index is persisted,and the memory calculation is used in use,which greatly improves the retrieval speed of the index data,and ensures the availability and fault tolerance of the index.Experiments show that the condition retrieval speed of the scheme is greatly improved compared with the original database,and the retrieval speed of the secondary index scheme based on Solr and HiBase is also improved.