摘要
网络爬虫是全文搜索引擎系统的核心模块,它实现对站内信息的检索和特定Web信息的搜索。然而,万维网规模越来越巨大,产生了专业爬虫。主要研究了怎样建立一个有效的专用爬虫,它可以自动发现和下载用户感兴趣的页面。包括的内容:超文本分类器、提取器、爬虫的基本功能、数据存储及核心算法。
The network spider is the nucleus module of full-text search engine system, which achieves the station-specific information retrieval and the specific Web information search. However, the specialized spider is produced while the World Wide Web scale is getting larger and larger. What this article studies is how to establish an effective special spider, which can automatically find and download pages users interested in. The pages include hypertext classifier, distiller, spider's basic function, data storage and core algorithm.
出处
《贵州师范大学学报(自然科学版)》
CAS
2009年第3期92-95,共4页
Journal of Guizhou Normal University:Natural Sciences