基于林产品贸易Web信息增量爬虫的研究

Research on Incremental Crawler Based on the Web Information of Forest Products Trade

随着互联网信息的快速增长和大数据掀起的热潮,如何保证主题数量的增量爬取再次成为获取信息的必要手段,增量爬虫能够以较少的资源实现信息的快速更新,从而提高信息检索的准确率。文章结合林产品贸易Web信息的特点,提出了一种适用于林产品贸易Web信息的基于优先级增量爬取方法,网页更新的优先级伴随着抓取而动态改变。最后通过实验证明随着抓取信息的增加该方法在保证网页质量的同时能有效的降低系统的资源使用,在海量信息爬取的过程中具有很好的应用价值。

With the rapid growth of internet information and the boom of big data,how to ensure the incremental crawling of theme amounts has become the main method to obtain information again,while increment crawling can achieve rapid update information with less resource and improve information retrieval accuracy.Combining with the characteristics of trade information in forest products website,this paper proposed the priority-based incremental method which changed the priority along with crawling.Finally,experimental results showed that this method could guarantee the quality of crawling pages when rapidly acquiring updated resource,and improve the accuracy of information retrieval,and then reduce resource usage of the system.