基于VSM与HITS融合的扩展主题型爬虫

An Extended Topic Crawler Based on the Fusion of VSM and HITS

目前主流开源爬虫框架在分析页面与主题领域关联性上,常采用基于关键词的量化和向量空间模型算法相融合,但融合疏忽了界面语义与特定主题间的关联,导致爬取内容与主题产生偏差。为了给金融等领域的舆情分析提供准确的数据支撑,提出一种面向领域扩展主题库的爬虫及系统,通过扩展主题特征库,融合向量空间模型(Vector Space Model,VSM)与超链接主题搜索算法(Hyperlink-Induced Topic Search,HITS),优化了主题页面相关度计算,并针对股票舆情信息爬取进行仿真。结果表明,上述扩展主题型爬虫在爬取准确率和效率等方面有较好地提升,能够有效地完成领域主题信息的爬取任务。

At present,mainstream open-source crawler frameworks often use keyword-based quantification and vector space model algorithms to analyze the relevance of pages and subject areas.However,the integration ignores the relationship between interface semantics and specific topics,which causes the deviation between crawling content and topics.In order to provide data support for public opinion analysis of finance and other fields,this paper proposes an extended topic crawler and system.By expanding the topic feature library and integrating the Vector Space Model(VSM)and the Hyperlink-Induced Topic Search(HITS)algorithm,it optimizes the relevance calculation of topic pages.Finally,the simulation experiments of crawling stock public opinion information show that the crawling accuracy and efficiency of extended topic crawlers have been improved and the crawler can accomplish the task of domain topic information acquisition effectively.