网络爬虫的设计与实现

The Design and Implementation of Web Crawler

搜索引擎技术随着互联网的日益壮大而飞速发展。作为搜索引擎不可或缺的组成部分,网络爬虫的作用显得尤为重要,它的性能直接决定了在庞大的互联网上进行网页信息采集的质量。设计并实现了通用爬虫和限定爬虫。

With the growing of Internet, search engine technology develops rapidly. As an indispensable part of search engine, web crawler is particularly important, its performance directly determines the quality of gathering webpage information in large Internet . This paper designs and implements general crawler and limitative crawler.