一种基于大数据技术的舆情监控系统

A Public Opinion Monitoring System Based on Big Data Technology

随着互联网的普及,社交网络已经成为人们生活中至关重要的一部分。这种新媒体时代的潮流促进了信息的流动和传播,同时也带来了海量的媒体内容与用户数据。社交媒体分析是舆情监控系统的主要内容,舆情数据的分析、处理与监控是新媒体时代带来的新的技术问题之一。近年来的大数据处理计算技术提供了处理海量数据的成熟解决方案。大数据处理平台有很多种,其中Hadoop具有成熟的社区,其架构稳定且易于使用。在文本分类方面,LDA统计模型方法给文本分类问题带来了新的处理办法。因此,文章提出了一个基于成熟开源架构的舆情监控系统,系统基于Hadoop平台,以Nutch作为爬虫,使用Solr实现核心的索引查找功能。整个平台在海量数据处理方面展现了较高的分析处理效率,在应对海量数据带来的问题的同时还提供了智能的分析与统计功能。

With the popularization of Internet, social network has become a vital part of people＇s lives. Social media promotes lfow and dissemination of information, but also brings a deluge of social media data and user data. Social media analysis is the main component of public opinion monitor system. Analysing and monitoring of public opinion data is one of the new technical problems caused by media in this era. In recent years, new technology such as big data processing provides proven solution to cope with the massive data . There are many big data processing platforms, in which Hadoop platform has a mature community and its structure is stable and easy to use. To text classiifcation problems, LDA statistical modeling brings a new approach. Therefore, this paper proposes a public opinion monitoring system based on proven open source architectures. The system bases on Hadoop platform, with Nutch as a crawler, using Solr to achieve the core index search function. The entire platform demonstrates its high efifciency in the mass data processing analysis. while providing intelligent analysis and statistical functions in response to the problems caused by massive amounts of data.