面向主题的Web信息收集系统的设计与实现

Design and Implementation of a Topic-focused Web Information-gathering System

随着互联网信息的持续爆炸性增长 ,通用搜索引擎的信息覆盖率和检索精度都在不断下降 ,发展面向主题信息的专用网络信息检索工具已经成为趋势 .文中提出的面向主题的Web信息收集系统是这类工具的核心部件 .该系统采用文档矢量模型进行文档相关度计算 ,并结合页面链接的上下文信息过滤页面 ;借鉴并修改了Shark启发式查找算法来查找相关页面 ;可采用多机并行下载提高收集效率 ;并依据站点的重要程度进行动态更新 .在一个面向Internet的计算机教学资源检索的搜索引擎中具体实现了这个Web信息收集系统 ,整个系统在低性能的台式机上就能运行 ,并可获得较高的属于指定主题的页面的收集精度和收集效率 .

With the explosive growth of information on Internet, the coverage and retrieval precision of mainstream search engines have been declining. It has become a tendency to develop specific Internet information retrieval tools. A topic-focused web information-gathering system described in this paper is the core facility of such tools. We use vector space Modal to represent and calculate the relevance of html web pages, filter web pages in conjunction with context of web pages link, search related pages using modified Shark-Algorithm, crawl web using multiple machines, update downloaded information according to the significance of web sites. We have implemented this information-gathering system in an Internet information search engine for computer educational resources. It works well on popular PC and achieves high topic-focused information retrieval precision and high crawl efficiency.