基于主题相关概念和网页分块的主题爬虫研究

Research on focused crawler based on topic-related concept and page segmentation

针对传统主题爬虫的不足,提出一种基于主题相关概念和网页分块的主题爬虫。先通过主题分类树获取主题相关概念集合,然后结合主题描述文档构建主题向量来描述主题;下载网页后引入网页分块来穿越"灰色隧道";采用文本内容和链接结构相结合的策略计算候选链接优先级,并在HITS算法的基础上提出了R-HITS算法计算链接结构对候选链接优先级的贡献。实验结果表明,利用该方法实现的主题爬虫查准率达66%、信息量总和达53%,在垂直搜索引擎和舆情分析应用方面有更好的搜索效果。

For the shortcomings of traditional focused crawler, this paper proposed a focused crawler based on topic-related concept and page segmentation. It set up topic vector by combining topic descriptive document with topic-related concept set which was generated by category tree to describe topic, and it introduced page segmentation after downloading a Web page to traverse grey tunneling. Then it took text content and link structure into consideration when computing the priority of candidate links. It also proposed a R-HITS algorithm based on the HITS algorithm to compute link structure＇ s contribution to priority of candidate links. The experimental result shows that, the precision of the focused crawler implemented by this method is 66% and sum of information is 53%. It has better effect on the applications of vertical search engine and public opinion analysis.