广域网分布式爬虫中的Agent协同与Web划分研究

Research on Agent collaboration and Web partition in WAN-based distributed Web crawlers

针对广域网环境下分布式Web爬虫的Agent协同和Web划分两个核心问题进行深入研究,提出了基于顾问服务的分布式Web爬虫系统模型,给出了详细的系统设计方案及Agent协同算法框架,并通过推导证明了顾问服务参与Agent协同能够使分布式爬虫系统承受相对较小的网络负载。提出了分布式Web爬虫Web划分的概念,围绕Web划分单元选取及Web划分策略,对Web划分的分类和实现进行了详细的讨论,并通过实验对多种Web划分方法进行了对比和评价,验证了广域网系统相对于局域网系统的优势,并发现运营商互连因素对爬虫系统性能的影响大于地理位置因素的影响。

This paper focuses on agent collaboration and Web partition, the two core issues in WAN-based distributed crawling. First, a new consultant-service-based agent collaboration method and the corresponding system model are proposed. The new method has a lower communication overhead than the central-coordinator-based crawling systems and exploits location proximity better than the ones based on Distributed Hash Table （DHT）. Second, the detailed definitions of Web partition are presented. The selection of Web partition unit and the Web partition strategy are discussed. The experiment under the real Interact environment shows that WAN-based distributed Web crawling systems have better performance than the LAN-based ones. The experiment also shows that the impact of Interact service providers interconnectivity on the system performance is greater than that of the geographical locality.