基于主题网络爬虫的不良网页的发现与识别

Unhealthy Webpage Detection Based on Topic-focused Web Crawler

针对互联网中出现的大量不良内容,分析出其主要特征,首次提出将不良网页的文本特征与搜索引擎中网络爬虫相结合的技术来主动寻找互联网中的不良网页及不良网站,并将结果分级别反馈到用户层以便对不良网页和网站进行处理,以达到净化网络环境的目的.实验结果表明,所提出的算法能够有效检测不良网页,并且能够很好地应对不良网站的反关键字过滤策略.

Internet is making massive amounts of harmful information, and it is very important to remove as much harmful information as possible to purify the internet. After the analysis of a large amount of harmful information on the internet, the key text features of harmful contents are presented. The novel approach is to find harmful Webpage and site by embedding the harmful text features into the Web spider of the search engine, and generate multi-level results to the users so that they can deal with the harmful Webpage and site to purify the internet environment. The experiments show that the proposed algorithm is capable to detect unhealthy Webpage effectively, and cope with the strategy of anti-keywords filtering from the unhealthy Website.