基于优先级队列的分布式多主题爬虫

Distributed multi-topic Web crawler based on priority queue

在分布式环境中,为提高资源利用率和网页抓取效率,提出一种基于优先级队列的分布式多主题爬虫调度算法PQ-MCSA。利用基于缓存的扩展式哈希算法对整体任务集进行切割,按照URL逻辑二级节点哈希映射法,将分割后的子任务集均匀地分配到各处理节点中;利用单处理节点的计算能力结合构建的任务优先级队列进行不同主题任务的调度。该算法改善了传统分布式爬虫对单节点的处理资源调度不充分、多主题任务爬取不均匀等缺点。实际项目的应用结果表明,使用该方法能够有效地提高各主题爬取结果的均衡度,具有较强的实用性。

To improve resource utilization and efficiency of Web crawling in a distributed environment ,a priority queue based scheduling algorithm for distributed multi‐topic crawler named PQ‐MCSA was proposed .The cache‐based extendible hashing al‐gorithm was utilized to cut the task set and then the divided tasks were distributed to each processing node evenly according to the URL secondary logical node hashing algorithm .The computing ability of single processing node and task priority queue previous‐ly constructed were combined to schedule different topic tasks .This algorithm overcomes shortcomings of the traditional distri‐buted crawler such as insufficient scheduling of single node resource and uneven crawling of multi‐topic tasks .The application in an actual project shows that this method can improve the balance degree of topic crawl results efficiently ,and has stronger practi‐cal value .