基于压缩倒排文件的中文全文检索仿真系统

Chinese Full-text Retrieval Simulation System Based on Compressed Inverted File

探讨基于压缩倒排文件的中文全文检索技术,包括数据压缩方法、存储、检索与排名机制。借助中科院的高精度ICTCLAS中文分词系统,采用C++/STL语言仿真实现了一个中文全文检索系统。该文列出部分关键代码,利用搜狗实验室提供的数据进行实验。通过改进压缩算法,系统的磁盘利用率提高了近80%。

This paper analyzes Chinese full-text retrieval technologies based on compressed inverted file, including data compression, file storage, searching and ranking mechanisms. A Chinese text retrieval simulation system is implemented in C＋＋/STL with ICTCLAS, which is a high precision Chinese segmentation tool from CAS. Some key codes are also included, and an experiment is carried using data provided by the Sogou Lab. The system disk utilization goes up nearly 80% through using improved compression algorithm.