Deep Web爬虫爬行策略研究

On research of deep web crawler's crawling strategy

如今Web上越来越多的信息可以通过查询接口来获得,为了获取某DeepWeb站点的页面用户不得不键入一系列的关键词集。由于没有直接指向DeepWeb页面的静态链接,当前大多搜索引擎不能发现和索引这些页面。然而,近来研究表明DeepWeb站点提供的高质量的信息对许多用户来说是非常有价值。这里研究了怎样建立起一个有效的DeepWeb爬虫,它可以自动发现和下载DeepWeb页面。由于DeepWeb惟一“入口点”是查询接口,DeepWeb爬虫设计面对的主要挑战是怎样对查询接口自动产生有意义的查询。这里提出一种针对查询接口查询自动产生问题的理论框架。通过在实际DeepWeb站点上的实验证明了此方法是非常有效的。

As an ever-increasing amount of information on the web today is available through search interfaces, users have to key in a set ofkeywords in order to access the pages from certain web sites, which are often referred to as the hidden web or the deep web. Since there is no static links to the hidden web pages, search engines cannot discover and index such pages. However, according to recent studies, the content provided by many hidden web sites is often of very high quality and can be extremely valuable to many users. How to build an effective hidden web crawler that can autonomously discover and download pages from the hidden web is studied. Since the only ＂entry point＂ to a hidden web site is a query interface, the main challenge to a hidden web crawler is how to automatically generate meaningful queries for issue to the site. A theoretical framework to investigate the query generation problem for the hidden web and we propose effective policies for generating queries automatically is provided. Experiment shows that these policies are effective.