基于遗传算法的主题爬虫策略改进

Improvement of Focused Crawling Strategy Based on Genetic Algorithm

针对主题爬虫存在"主题漂移"的问题,为了快速抓取网页,提出了一种基于遗传算法的主题爬行改进策略。在现有遗传算法爬行策略的基础上新引入了PageRank算法,调整了以往网页主题相关度计算方式,以计算得到的网页PageRank和相关度值为依据来选择爬行中的遗传因子,并重新设置了适应度函数,在保证优越遗传因子(与主题相关且重要网页)被优先遗传的同时,减少了遗传因子在传递过程中的"主题漂移",从而使爬行到网页的重要性和主题相关性均有所提高。与以往基于遗传算法的策略相比较,在不影响查全率的情况下,与主题相关且重要的网页数可提升5%以上。

Aiming at the subject drifting problem of topic crawling,this paper presents an improved strategy.Based on Genetic Algorithm,this strategy absorbs the idea of the PageRank algorithm and correlation of page,re-sets the fitness function and adjusts the size of correlation parameters of page by it.In this way,the superior gene is selected first and the subject drifting is reduced while delivering.Compared with previous strategies based on genetic algorithm,without prejudice to recall the circumstances,the number of pages relevant to the subject can raise more than 5%.