面向主题网络爬行的智能隧道穿越算法研究

Focused crawling oriented intelligent tunneling algorithm research

在深入研究传统的主题网络爬虫爬行策略的基础上,提出了一种全新的智能隧道穿越算法。该算法将快速在线Q(λ)学习算法引入普通隧道技术中,通过在线学习积累爬行过程中的经验,引导主题网络爬虫以最小的时空代价穿越主题相关度低的区域到达主题相关度高的区域。模拟实验表明,与普通隧道相比,智能隧道算法降低了时空复杂度,提高了主题网络爬虫的性能与运行效率。

This paper proposed a new intelligent tunneling algorithm based on deep research on traditional focused crawling strategy. This algorithm combined with tunneling technology and fast online Q （ A ） learning and led focused crawler to go through from low relevant areas to high ones with the smallest cost by online learning and experience accumulation. In the simulation experiment, compared intelligent tunneling algorithm with normal tunneling, the result shows that the intelligent tunneling algorithm has reduced the time space complexity and improves performance and increases efficiency of focused crawler.