基于关键词相关度的Deep Web爬虫爬行策略

Crawlers Crawling Strategy of Deep Web Based on Keywords Relevant Weight

Deep Web蕴藏丰富的、高质量的信息资源,为了获取某Deep Web站点的页面,用户不得不键入一系列的关键词集。由于没有直接指向Deep Web页面的静态链接,目前大多数搜索引擎不能发现这些页面。该文提出的Deep Web爬虫爬行策略,可以有效地下载Deep Web页面。由于该页面只提供一个查询接口,因此Deep Web爬虫设计面对的主要挑战是怎样选择最佳的查询关键词产生有意义的查询。实验证明文中提出的一种基于不同关键词相关度权重的选择方法是有效的。

There is plenty high-quality information in Deep Web, but user has to input several keywords to search and reach the pages of Deep Web. Traditional crawlers cannot get to the Hidden Web pages because there are no direct links to pages of Deep Web. This paper presents a crawling strategy that can download the pages of Deep Web effectively. As the result of the only interface that Deep Web provides, the biggest challenge for Deep Web crawler is how to choose the best keywords to query effectively. This paper brings forward a new selecting method that based on the relevant weight of different keywords. The experiment shows that this method is efficient.