基于爬虫技术和语义分析的网络舆情采集系统设计

Design of online public opinion collection system based on crawler technology and semantic analysis

为满足当前高校学生舆情监督的工作需要,针对传统网络舆情采集系统的舆情信息采集效率低下、质量差等问题,设计和开发了一套基于爬虫技术与语义分析的网络舆情采集系统。该系统通过主题网络爬虫对互联网上感兴趣的资源数据进行爬取,同时利用语义分析中的向量空间模型(VSM)算法对采集到的网页数据进行进一步过滤,充分保证所采集到的舆情数据的质量。系统的各项性能试验测试结果表明,文中所设计的系统能够按照用户需求在保证效率和质量的情况下,较好地完成互联网舆情信息采集,验证了系统的可行性。

In order to meet the needs of the supervision of college students’public opinion,aiming at the problems of low efficiency and low quality of traditional online public opinion collection system,designs and develops a network public opinion collection system based on crawler technology and semantic analysis.The system crawls the interested resource data on the Internet through the topic Web crawler,and further filters the collected web page data by using the Vector Space Model(VSM)algorithm in semantic analysis to fully ensure the quality of the collected public opinion data.The performance test results of the system show that the system designed in this paper can complete the collection of public opinion information in the Internet better according to the user’s needs and ensure the efficiency and quality,which verifies the feasibility of the system.