网页去重在基于Web企业竞争情报平台中的应用与研究

The Application and Research of Duplicated Web Pages Based on the Web Platform of the Competitive Intelligence

互联网上大量重复网页的存在,严重地影响信息的检索质量.因此提出了一种基于特征码散列的网页去重算法,利用特征码对特征语句集散列以缩小其比较的范围,提高了网页去重的准确性.实验表明该算法准确率高,性能良好,基于上述算法实现了基于Web的企业情报竞争平台中的网页去重.

A large number of duplicated Web pages are on the internet, which has seriously affected the quality of information retrieval. Therefore, the paper has proposed an algorithm based on the signature hashing of the duplicated web pages, using the distributed signature characteristics of the sentences to narrow the relative scope of the pages and to improve the accuracy. The experiment result shows the high accuracy rate and good performance of the algorithm, which helps realize the goal based on the web platform of business competitive intelligence.