Lucene索引段合并优化策略

Merging optimization strategy of Lucene’s index segment

随着大数据应用发展,如何从海量数据中进行高效信息搜索成为研究热点。Lucene全文搜索引擎通过索引段合并来提高索引效率,但Lucene索引段合并过程大多需要从磁盘加载各索引段,将占用大量系统资源,降低系统吞吐量。针对该问题,提出基于Lucene索引段合并优化策略,该策略通过负载系数来选择不同的索引段合并操作。为提高数据的检索速度,进一步建立索引段相似度评价模型来选择出最优合并索引段集合进行合并。通过与现有Tiere,LogByte,LogDoc等合并策略进行实验对比,提出的优化策略能有效减少索引段合并次数,提升系统吞吐量及索引效率。

With the development of big data applications,how to search efficient information from massive data has become the hotspot of research.Lucene,which is a full-text search engine,improves index efficiency by the mechanism of merging index segment.However,since the merging process of index segment mostly needs to load each index segment from disk,it will consume a lot of system resources and reduce system throughput.Aiming at this problem,a merging optimization strategy based on Lucene index segment is proposed.The strategy considers the node load to select the merging operation of index segment.In order to improve the retrieval speed of data,an index segment similarity evaluation model is further established to select the optimal index segment to be merged.By comparing with Tiere,LogByte,LogDoc and other merge strategies,the proposed optimization strategy can effectively reduce the number of merges of index segments and improve system throughput and index efficiency.