基于网页分块技术主题爬行器的实现

Realization of Focused Crawler Based on Page Segmentation

针对目前通用搜索引擎搜索到的结果过多、与主题相关性不强的现状,提出一种基于网页分块技术的主题爬行器实现方法,并实现了一个原型系统Crawler1.实验结果表明,本系统性能较好,所爬网页的相关度在55%以上.

In the light of result returned currently by general-purpose search engines being excessive, and having no strong similarity with the topic, this paper covers a technique of dividing the web page to chunks to implement a focused crawler. With this method, Crawlerl, a prototype of a focused crawler has been realized. Experimental results indicate that Crawlerl has better performance. The number of topic web pages crawled by Crawlerl attains more than 55%.