一个Web抓取器的实现

An Implementation of Web Grabber

本文介绍一个Web抓取器的实现。Web抓取器能够根据用户输入的URL和一定的约束条件自动地搜索Web上的超文本文档 ,建立URL索引 ,根据该索引有选择的下载Web资源 ,同时保持了原有的链接关系。本文实现了一种Web遍历过程 ,并根据遍历结果抓取文档 ,从而完成对指定URL抓取文本。

This paper introduces an implementation of Web grabber. The primary purpose of the Web grabber is to fetch hypertext documents according to the given URL and some restrictions. After specifying the options, our program will fetch the start Web document, extract the links from document.Having traveled the URLs,it will finally download the hyperlinked documents.The authors analyse an implementation of such a Web grabber. After presenting a scheme of Web crawling,this paper describes the details of fetching Web documents according to the result of crawling . [