绿色网络网页正文内容提取算法

Text Content Extraction Algorithm for Green Network Webpage

互联网中的网页有较多商业广告,绿色网络系统无法过滤其中具有不良内容的网站。为解决该问题,提出一种绿色网络网页正文内容提取算法。通过文件对象模型树识别与提取网页正文内容模块,使用基于粒子群的权值优化算法对网页正文各个板块特征权值进行评分,利用与不良关键字的比较,确定并过滤不良网页。实验结果表明,经粒子群权值算法优化提取后,绿色网络系统对不良网页的识别准确率为86.9%,召回率为95.6%,F值为91.02%,比优化前有较大提高。

At present, the Web pages have more business in the Internet advertising, the green network system can not filter the site with poor content. In order to solve this problem, this paper proposes a text content extraction algorithm for green network webpage. It uses the Document Object Model（DOM） tree to identify and extract the pages of text content module, uses an optimized content extraction algorithm based on particle swarm weight to score each section of the main content, compares the scores with the unhealthy keywords to identify and filter harmful Web pages. Experimental results show that, after optimized by new algorithm, the accuracy rate of identifying harmful webpage is 86.9%, the recall rate is 95.6%, the F value is 91.02%, and is higher than before optimization.