基于网页空间进化算法的暴雨灾害主题爬虫策略

Focused Crawler for Rainstorm Disaster Strategy Based on Web Space Evolutionary Algorithm

针对单目标优化算法求解爬虫问题时难以获得最优加权因子和易于陷入局部最优的缺点,将多目标优化算法引入主题爬虫,提出一种基于多目标优化的网页空间进化算法。通过计算测试链接与种子链接库中链接的最短距离,将其与种子链接库中所有链接间的平均距离进行比较来更新种子链接库。针对多目标优化中Pareto最优解的选取问题,给出一种最近最远候选解法。实验结果表明,与宽度优先搜索等算法相比,该算法具有较高的爬准率和稳定性。

Aiming at the shortcomings of single target optimization algorithm to solve the problem that the crawler problem is difficult to obtain the optimal weighting factor and easy to fall into the local optimum,the multi-objective optimization algorithm is introduced into the topic crawler,and a Web Space Evolution (WSE) algorithm based on multi-objective optimization is proposed.The seed link library is updated by calculating the shortest distance between the test link and the link in the seed link library,comparing it to the average distance of all links in the seed link library.Aiming at the selection of Pareto optimal solution in multi-objective optimization,a recent farthest candidate solution is proposed.Experimental results show that compared with the algorithm of breadth-first search,the algorithm has high tracking rate and stability.