摘要
现存主题爬虫算法在抓取主题网页方面,其准确性不是很高。本文提出一种基于文本内容评价与网页链接评价的主题网页抓取方法。首先计算当前网页与主题的相关度,然后将相关度值与给定阈值进行比较决定当前网页是丢弃还是存储,同时相关度值的大小也决定了待爬链接队列中URL的优先权,此模型考虑了主题网页的准确率与覆盖率之间的平衡。新设计的主题爬虫算法在抓取主题网页方面,其准确性有一定程度的提高。
In the aspect of grasping the topical webpage to the existing topical crawler algorithm, its accuracy is not high. This paper presents a topical webpage grasping method which based on evaluation of text content and webpage link. First it calculates the correlation of current webpage and theme, and then compares the correlation values with a given threshold to determine the current webpage is discarded or stored. At the same time the size of the correlation value also determines the priority of URL in the climbing link queue, this model takes into account the balance of topical webpage between accuracy and coverage. In the as- pect of grasping topical webpage to design the new topical crawler algorithm, its accuracy has been improved to some extent.
出处
《计算机与现代化》
2014年第4期77-80,共4页
Computer and Modernization
关键词
主题爬虫
主题相关度
主题网页
topical crawler
topical correlation
topical webpage
