网页去重方法研究及算法设计

Study on duplicated removal method Web pages and algorithm design

针对在网页检索结果中经常出现内容相同的冗余页面的问题,为了按用户偏好对搜索结果进行筛选,提出了一种通过中文字主题要素学习网页内容的网页去重算法.该算法通过抽取网页要素中用户偏好的短语来抽取网页的内容,根据学习的网页内容计算其相似度来判断网页的重复度.经与相似方法比较,结果表明,该方法能够完成针对中文内容网页的去重,具有较高的查全率和查准率.

In view of the Web page retrieval result, the user frequently obtains the same content redundant page. In order to solve this problem, one kind of duplicated Web pages removal algorithm according to the user＇s interest is proposed to study Web pages content on elements of Chinese topic. This method follows the three steps ： First, extract phrase of the user＇s interest which is Chinese elements content of Web pages, then study Web pages content extracted and finally, calculates their similarity according to the study news content to judge the heavy multiplicity of the news homepage. Compared with simliar methods, the experimental result indicated that, in view of the Chinese content web pages duplicated, this method obtains the high recall and the accuracy ratio.