深度搜索中下一页链接获取策略的研究

The Research of Finding Next Page Hyperlink Strategy in Deep Search

高质量、结构化的数据对很多应用来说其价值是无穷的。然而这些数据往往都隐藏在查询表单背后的深度Web数据库中。深度Web数据库中的内容不能够直接被爬虫通过超链接直接获取,只有通过主动对查询表单提交查询才能有效地获取这些数据。最近的研究主要针对以下两个方面:一是如何有效的理解和获取网络上的查询表单,二是如何不断的对查询表单进行查询并下载表单背后的资源。一个必要的但还没很好的解决的一个问题是:如何将一次查询所获得的所有结果页全部有效的获得?这个问题从本质上看就是如何获取结果页面中"下一页"超链接的问题。本文针对这一问题提出了一种通过分析"下一页"超链接的结构特征来识别"下一页"超链接的策略,并在多种情况下进行了试验,取得了良好的效果。

The high quality, structured data is invaluable for many applications. These data always hidden in the Deep Web databases. Deep Web databases are not directly crawled by Web crawlers through hyperlinks and are only accessible through Web query forms interfaces. Recent research efforts have been focusing on two aspects： one is to understand these Web query forms, the other is to efficiently acquire the structured information inside Web databases through iteratively issuing meaningful queries. A critical but still largely unresolved question is： how to acquire all the pages from one query？ And this question esseni^e turns to how to find the ＂next page＂ hyperlinks. In this paper we focus on this issue and bring forward a strategy that find the ＂next page＂ hyperlinks from the structure of the ＂next page＂ hyperlinks. We do experiments in many cases, and the results seem good.