主题网络爬虫研究综述

Survey on topic-focused crawlers

网络信息资源呈指数级增长,面对用户越来越个性化的需求,主题网络爬虫应运而生。主题网络爬虫是一种下载特定主题网页的程序。利用在采集页面过程获得的特定信息,主题网络爬虫抓取的页面都是与主题相关的。基于主题网络爬虫的搜索引擎以及基于主题网络爬虫构建领域语料库等应用已经得到广泛运用。首先介绍了主题爬虫的定义、工作原理;然后介绍了近年来国内外关于主题爬虫的研究状况,并比较了各种爬行策略及相关算法的优缺点;最后提出了主题网络爬虫未来的研究方向。

With the exponential growth of network information resources and the growing personal- ized demands of customers, topic-focused crawler emerges as the times require. Topic-focused crawlers are programs designed to download web pages which are relevant to specific topics. Using information gathered at running time, topic-focused crawlers explore the webs which follow promissory hyperlinks, and fetch only pages which appear to be relevant. The searching engine and corpus building based on topic-focused crawling have been widely used. We first define the goals and operating principles of focused crawling, comprehensively analyze the recent advances at home and abroad, and then compare the crawling strategies of various topic-focused crawlers as well as the advantages and disadvantages of related algorithms. Finally, we point out the future direction of topic-focused crawling.