大规模非结构化数据的索引技术研究

Study on Large-scale Unstructured Data Indexing Technology

为解决搜索引擎ASPSeek在大规模数据下检索效率低下、占用空间大以及不利于更新等问题,提出了一种分块式存储的倒排索引组织技术,并对基于外存的B+树索引和线性散列索引的性能进行了比较测试研究。测试结果表明,查询每万条数据耗时线性散列为B+树索引快57.40%,插入每万条数据耗时线性散列为B+树索引的2.44倍,删除每万条数据耗时线性散列为B+树索引的83.52%,线性散列索引文件大小为B+树索引文件大小的109.56%。由测试结果可知,B+树索引具有较快的索引构建和更新速度,而线性散列索引则具有较高的磁盘空间占用率和较好的查询性能。

To solve the problem that in large- scale data condition the ASPSeek search engine retrievals inefficiently,has large disk space occupancy and can＇t be conducive to update,propose an inverted index- organized technique based on block storage,and make a performance comparison research test between external memory based B ＋ tree index and linear hash index. Test results showthat,for queries per million data- consuming linear hashing to B ＋ tree index is 57. 40%,for inserting per million data- consuming linear hash is 2.44 times to B ＋ tree index,for deleting every million data- consuming linear hash to B ＋ tree index is 83. 52%,linear hash index file size is 109. 56% of B ＋ tree index file size. According to the test results,B ＋ tree index has the faster index building and updating speed,while linear hash index has the higher disk space occupancy rates and better query performance.