Deep Web爬虫的一种增量式更新策略

An Incremental Update Strategy of Deep Web Crawler

为了在网络及计算机硬件等资源有限的前提下,最大限度地保证本地副本的"新鲜"程度,往往需要为Deep Web爬虫定制一个重爬机制。为了满足这种需求,提出了一种基于采样的增量式Deep Web更新方法,使用贪心策略分配爬虫资源,以最大限地的提高资源利用率。经实验验证,该方法取得了比较优异的结果。

In the context of network and computer hardware resources,in order to to ensure the maximum ＂fresh＂ level of local copy,we need to design an unpdate method for crawler.To meet the demands,we propose a sampling based incremental Deep Web update method.This method uses the greedy strategy to allocate crawler resources for achieving maximum resource utilization.We verify this method experimentally to obtain excellent results comparation.