基于模式匹配的网页净化方法

Eliminating Noisy Information in Web Pages Based on Pattern Matching

新闻网页主要由大量文字描述构成,相比网页其他区域的噪音内容,其主题内容含有大段连贯的文字。根据这一特点提出一种基于模式匹配的网页净化方法,即在网页源代码中匹配最长文字字符串,从而准确定位主题内容源代码在网页源代码中位置,实现网页净化。本方法可去除来自不同网站网页的噪音内容,无需事先训练数据集来生成模板,不需要生成网页DOM树。对同构、异构和不符合XML规范的网页净化,试验证明效果理想且性能稳定。

A piece of news web page has a lot of words.Compared with the noisy content,the topic content contains larger sections of coherent text.According to this feature,we propose an approach to purify pages based on pattern matching.This searches a page＇s source code for the location of the longest text string.The longest text string belongs to the topic content.That means we get the location of the topic content.It is a simple and fast approach for isomorphic pages,non-isomorphic pages and pages not meeting XML specification.The test proves it is satisfactory and stable.