摘要
针对搜索引擎在海量数据中搜索速度慢,占用存储空间大,对重复的网页去重性差的现状,提出一种基于Rabin指纹算法的去重方法,不仅对搜索到的URL地址进行去重,还对非重复URL地址对应的网页内容进行相似和相同的去重,试验表明能有效地提高搜索速度、节省存储空间,增强搜索的精度.
The existing search engine of massive data takes up large memory, needs much time and provides results of great duplication rate. To overcome these disadvantages, this paper proposes a duplication removal method based on the Rabin Fingerprint method, which cannot only remove the duplicated URL, but also remove the same even similar website content on different URL so that it can speed up the searching speed, save the memory capability and improve the accuracy of the research.
出处
《计算机系统应用》
2015年第7期128-131,共4页
Computer Systems & Applications
基金
国家档案局项目(2014-X-65)
