基于Web的专用爬虫的研究

The research and design of the specialized spider base on the Web

网络爬虫是全文搜索引擎系统的核心模块,它实现对站内信息的检索和特定Web信息的搜索。然而,万维网规模越来越巨大,产生了专业爬虫。主要研究了怎样建立一个有效的专用爬虫,它可以自动发现和下载用户感兴趣的页面。包括的内容:超文本分类器、提取器、爬虫的基本功能、数据存储及核心算法。

The network spider is the nucleus module of full-text search engine system, which achieves the station-specific information retrieval and the specific Web information search. However, the specialized spider is produced while the World Wide Web scale is getting larger and larger. What this article studies is how to establish an effective special spider, which can automatically find and download pages users interested in. The pages include hypertext classifier, distiller, spider＇s basic function, data storage and core algorithm.