基于Hadoop和HBase的分布式索引模型的研究

Research of distributed indexing model based on Hadoop and HBase

随着互联网上信息量飞速增长,海量数据的索引出现了难题,现行的索引方案已经难以提供高效、可靠的服务,为此,设计并实现了一种针对海量数据进行索引的平台模型。该平台模型首先利用Solr分布式索引器生成索引文件,然后利用Hadoop分布式集群,以HDFS分布式文件系统、Map Reduce分布式并行计算模型、Zookeeper同步协同系统以及Hbase分布式数据库技术来处理、协调管理索引和存储海量数据,最后通过实验测试,该平台模型可以克服现行的海量数据索引时存在的效率低的问题,同时具有良好的扩展性和可靠性。

With the rapid growth of the amount of information on the Internet,massive data index appeared,the current index scheme has been difficult to provide efficient and reliable service,therefore,designing and implementing an index for massive data platform model. The platform model using Solr distributed index is generated index file,and then uses the Hadoop cluster,HDFS distributed file system,Map Reduce distributed computing model, Zookeeper synchronous collaborative system and Hbase distributed database technology to handle the coordination management,indexing and storage of mass data,finally,through the experimental test,the platform model can overcome the existing efficiency of mass data index of low current problems,and has good expansibility and reliability.