基于视觉特征的网页信息抽取方法研究

Research on Web Page Information Extraction Based on Visual Features

面对大规模异构网页,基于视觉特征的网页信息抽取方法普遍存在通用性较差、抽取效率较低的问题。针对通用性较差的问题,该文提出了基于视觉特征的使用有监督机器学习的网页信息抽取框架WEMLVF。该框架具有良好的通用性,通过对论坛网站和新闻评论网站的信息抽取实验,验证了该框架的有效性。然后,针对视觉特征提取时间代价过高导致信息抽取效率较低的问题,该文使用WEMLVF,分别提出基于XPath和基于经典包装器归纳算法SoftMealy的自动生成信息抽取模板的方法。这两种方法使用视觉特征自动生成信息抽取模板,但模板的表达并不包含视觉特征,使得在使用模板进行信息抽取的过程中无需提取网页的视觉特征,从而既充分利用了视觉特征在信息抽取中的作用,又显著提升了信息抽取的效率,实验结果验证了这一结论。

Facing with the large-scale heterogeneous web pages, web extraction methods based on visual features tend to have poor generality and low extraction efficiency. To deal with the issue of poor generality, this paper proposes WEMLVF,a Web page information extraction framework based on visual features using supervised machine learning. This framework has good versatility. The effectiveness of the framework is validated through experiments on forum sites and news review sites. Then,to deal with the issue of low efficiency,the framework WEMLVF is utilized and method is proposed for automatically generating information extraction templates based on XPath and SoftMealy (a wrapper induction algorithm). These two methods use visual features to automatically generate information extraction templates without visual features. It makes full use of visual features information extraction and significantly improve the efficiency of information extraction,which is empirically verified.