高性能文本索引系统的设计与实现

Design and implementation of high-performance text index system

针对传统文本索引技术空间消耗大、分词不准确等问题,设计并实现了高性能文本索引系统。该系统采用压缩的全文自索引算法,节省了空间开销,避免了自然语言分词方法的影响,配合通配符搜索算法扩展了模糊搜索的功能,在众核CPU高性能处理器上可实现多线程并行处理,提高了处理速度,整个系统的实现是基于Web方式的,可以跨平台运行。实验结果表明,该系统将文本索引的空间消耗降为原文本的50%左右,具有较高的实用价值。

The traditional text index technology has problems like oversize space consumption,inaccurate word segmentation,etc. In order to solve these problems,a high-performance text index system is designed and implemented.This system employs com-pressed full-text self-index algorithm,which helps save space and avoid influence from natural language word segmentation.Com-bined with wildcard search algorithm,its search function is enriched.And applied on high-performance processors with many-core CPU,this system can execute in a parallel multi-thread way,which increases processing speed.The entire system is Web-based,so that it can be run in different operating systems.Experimental results show that this systemhas high practical value in thatit reduces space consumption to about 50%of primary text.