基于布局特征与语言特征的网页主要内容块发现

A WebPage Content Block Detection Method Based on Layout Features and Language Features

本文综合分析了网页内容块各方面的特征,提出了一个联合使用布局特征和语言特征的网页主要内容块发现方法,有效地解决了以往模型中通用性与高准确率不能共存的缺点。该方法使用网页视觉块树表示网页,对网页内容块的布局特征和语言特征分别建立了独立的分类器,然后组合这两个分类器来进行网页内容块分类。实验结果表明,在保持非噪音块召回率在90%以上的同时,组合分类器的准确率达到85%,比只使用布局特征的分类器提高5个百分点,比只使用语言特征的分类器提高15个百分点;在5个站点上的分类结果表明组合分类器在不同站点上性能稳定,具有良好的通用性。

This paper analyzed the different feature types of webpage blocks, and presented a webpage content block detection rnethod based on layout features and language features, which effectively resolved the seesaw problern be-tween detection accuracy and model generality across different types of webpages. The method used the vision-block tree to represent webpage, built two individual classifiers respectively for webpage＇s layout features and language features, and used different strategies to combine these two classifiers. The experimental results show that, with holding the content block detection recall higher than 90%, the combined classifiers＇ accuracy can reach 85 per-cents, 5 percents higher than the classifier using only the layout features, and 15 percents higher than the classifier using only the language features; and the experimental results also show that the combined classifiers obtained good detection performance over five selected websites which means that it have good generality.