基于本体的网络爬虫技术研究

A Study of Ontology-based Web Crawler

互联网已经成为最大的非结构化数据库,极大方便了信息访问.然而,网络上的信息大多都是无组织的,由于网络的分布式特性,很难对它进行信息和知识管理.因此,如何建立一个智能的信息发现机制很有必要.本文在分析了爬虫工作原理和传统算法后,提出了一种基于本体的网络爬虫的信息发现框架.该框架包含了预处理模块和本体管理模块,定义了网页相关度计算策略,最后通过实验对该框架进行了评估.

The Web, the largest unstructured database of the world, has greatly improved access to information. However, information on the Web is largely disorganized. Due to the distributed nature of the World Wide Web it is difficult to use it as a tool for information and knowledge management. Therefore, user doing the difficult task of exploring the Web has to be supported by intelligent means. This paper proposes an approach for information discovery building on a comprehensive framework for ontology-based web crawler. Our framework includes preproeessing module and ontology management module. It defines a relevance computation strategies of the web page and provides an empirical evaluation which has shown premising results.