BBS舆情系统爬虫模块的研究

Research on crawler module of BBS Public Opinion System

针对重要的舆论平台BBS论坛,提出利用网络爬虫构建适合抓取论坛数据的爬虫模块。通过分析BBS论坛以及比较不同类别爬虫的特点,介绍BBS爬虫模块的工作流程以及需要选择的策略,讨论URL的处理与重定向问题的解决方法,并提出对不同论坛模块定制不同任务的抓取策略。

In this paper,a web crawler module using web crawler was proposed to acquire large collections of web pages in the BBS.Based on the characteristics of BBS and the differences between crawlers,this work described the workflow and work strategy of the crawler module.The process and redirection of the URL was discussed.Furthermore,the idea of using different strategy to collect pages from different part of the BBS was also proposed.