基于文本块密度与标签路径等特征的正文提取

Text Extraction Based on Text Block Density with Tag Path and Other Features

为了解决网页中除正文信息外还包含网页导航、广告和免责声明等噪声信息的问题,本文提出一种基于标签路径等多特征和文本块密度的正文提取方法.首先根据文本块密度特征确定正文区域,然后在区域内使用标签路径等特征剔去噪音节点,最后抽取该文本块中的正文节点内容.该方法有效解决了网页正文块中噪声信息难以过滤和标签路径等特征易对正文部分外较长文本误抽取的问题,且无须训练和人工处理.从知名网站上随机选取新闻网页数据集进行实验,验证了该方法在不同数据源上都具有很好的适用性,抽取精确度优于CETR、CETD等方法.

Most of web pages contain content information as well as a lot of noisy information.In order to address this problem and improve the accuracy of web page extraction,a web page extraction method is proposed via text block density with tap path and other features.The proposed method mostly combines the advantages of text block extraction method and label path extraction method.First,the block of the text is determined according to the density feature of the text block,and then the tag path method is used to remove the noisy node in the block,the text node in the text block is extracted from the content finally.This solution effectively solves the problem that the noisy information in the text block is difficult to filter and the tag path method is easy to extract the long text from the noisy block.In the end,experiments show that the solution is better than CETR and CETD in most cases.