Big Data Stream Analytics for Near Real-Time Sentiment Analysis

In the era of big data, huge volumes of data are generated from online social networks, sensor networks, mobile devices, and organizations’ enterprise systems. This phenomenon provides organizations with unprecedented opportunities to tap into big data to mine valuable business intelligence. However, traditional business analytics methods may not be able to cope with the flood of big data. The main contribution of this paper is the illustration of the development of a novel big data stream analytics framework named BDSASA that leverages a probabilistic language model to analyze the consumer sentiments embedded in hundreds of millions of online consumer reviews. In particular, an inference model is embedded into the classical language modeling framework to enhance the prediction of consumer sentiments. The practical implication of our research work is that organizations can apply our big data stream analytics framework to analyze consumers’ product preferences, and hence develop more effective marketing and production strategies.

Sentiment Drift Detection and Analysis in Real Time Twitter Data Streams

Handling sentiment drifts in real time twitter data streams are a challen-ging task while performing sentiment classifications,because of the changes that occur in the sentiments of twitter users,with respect to time.The growing volume of tweets with sentiment drifts has led to the need for devising an adaptive approach to detect and handle this drift in real time.This work proposes an adap-tive learning algorithm-based framework,Twitter Sentiment Drift Analysis-Bidir-ectional Encoder Representations from Transformers(TSDA-BERT),which introduces a sentiment drift measure to detect drifts and a domain impact score to adaptively retrain the classification model with domain relevant data in real time.The framework also works on static data by converting them to data streams using the Kafka tool.The experiments conducted on real time and simulated tweets of sports,health care andfinancial topics show that the proposed system is able to detect sentiment drifts and maintain the performance of the classification model,with accuracies of 91%,87%and 90%,respectively.Though the results have been provided only for a few topics,as a proof of concept,this framework can be applied to detect sentiment drifts and perform sentiment classification on real time data streams of any topic.

Integrated Real-Time Big Data Stream Sentiment Analysis Service

Opinion (sentiment) analysis on big data streams from the constantly generated text streams on social media networks to hundreds of millions of online consumer reviews provides many organizations in every field with opportunities to discover valuable intelligence from the massive user generated text streams. However, the traditional content analysis frameworks are inefficient to handle the unprecedentedly big volume of unstructured text streams and the complexity of text analysis tasks for the real time opinion analysis on the big data streams. In this paper, we propose a parallel real time sentiment analysis system: Social Media Data Stream Sentiment Analysis Service (SMDSSAS) that performs multiple phases of sentiment analysis of social media text streams effectively in real time with two fully analytic opinion mining models to combat the scale of text data streams and the complexity of sentiment analysis processing on unstructured text streams. We propose two aspect based opinion mining models: Deterministic and Probabilistic sentiment models for a real time sentiment analysis on the user given topic related data streams. Experiments on the social media Twitter stream traffic captured during the pre-election weeks of the 2016 Presidential election for real-time analysis of public opinions toward two presidential candidates showed that the proposed system was able to predict correctly Donald Trump as the winner of the 2016 Presidential election. The cross validation results showed that the proposed sentiment models with the real-time streaming components in our proposed framework delivered effectively the analysis of the opinions on two presidential candidates with average 81% accuracy for the Deterministic model and 80% for the Probabilistic model, which are 1% - 22% improvements from the results of the existing literature.

Quality control of marine big data——a case study of real-time observation station data in Qingdao

Offshore waters provide resources for human beings,while on the other hand,threaten them because of marine disasters.Ocean stations are part of offshore observation networks,and the quality of their data is of great significance for exploiting and protecting the ocean.We used hourly mean wave height,temperature,and pressure real-time observation data taken in the Xiaomaidao station(in Qingdao,China)from June 1,2017,to May 31,2018,to explore the data quality using eight quality control methods,and to discriminate the most effective method for Xiaomaidao station.After using the eight quality control methods,the percentages of the mean wave height,temperature,and pressure data that passed the tests were 89.6%,88.3%,and 98.6%,respectively.With the marine disaster(wave alarm report)data,the values failed in the test mainly due to the influence of aging observation equipment and missing data transmissions.The mean wave height is often affected by dynamic marine disasters,so the continuity test method is not effective.The correlation test with other related parameters would be more useful for the mean wave height.

符合Real-time CORBA规范的传感器网络监控系统设计

传感器网络在解决底层数据源和I/O设备的异构性、处理连续查询的实时性、服务质量自适应等方面提出了许多新的挑战。提出一种基于实时CORBA的传感器网络数据流处理框架,说明了符合实时中间件规范的数据流处理系统设计的方法和特点,通过实验对该框架进行了性能测试,验证了其处理连续查询的实时性。

基于Spark Streaming的气象自动站实时流处理与存储系统

在当前大数据技术蓬勃发展的时代,人们对气象数据的实时处理、数据质量、数据存储及大规模查询等要求也越来越高。针对现有气象自动站数据业务落地环节多,任务处理耦合紧但系统部署分散等问题,文中基于Spark Streaming的流式计算框架,研究使用Flume解析收集自动站原始数据,在Spark Streaming中设计融入自动站数据质控算法,最终通过对分布式数据库存储的表设计,使气象自动站数据具备高效率、高质量、高可靠的应用服务能力。性能测试结果表明,基于Spark Streaming的气象自动站数据实时流处理与存储系统,数据从文件采集、解码、流处理至入库的全流程能够在秒级完成,TB级数据查询响应为毫秒级,加权查询为秒级,完全满足自动站数据业务应用需求,从而为进一步提高气象自动站数据质量与服务水平提供基础支撑。

Events Sourcing and Command Query Responsibility Segregation Based Fast Data Architecture

With the advent of Big Data, the fields of Statistics and Computer Science coexist in current information systems. In addition to this, technological advances in embedded systems, in particular Internet of Things technologies, make it possible to develop real-time applications. These technological developments are disrupting Software Engineering because the use of large amounts of real-time data requires advanced thinking in terms of software architecture. The purpose of this article is to propose an architecture unifying not only Software Engineering and Big Data activities, but also batch and streaming architectures for the exploitation of massive data. This architecture has the advantage of making possible the development of applications and digital services exploiting very large volumes of data in real time;both for management needs and for analytical purposes. This architecture was tested on COVID-19 data as part of the development of an application for real-time monitoring of the evolution of the pandemic in Côte d’Ivoire using PostgreSQL, ELasticsearch, Kafka, Kafka Connect, NiFi, Spark, Node-Red and MoleculerJS to operationalize the architecture.

基于流计算和大数据平台的实时交通流预测

目前交通流预测实时性差,很难满足在线分析和预测任务的需求,基于此提出一种Flink流计算框架和大数据平台结合的实时交通流预测方法。基于流计算框架实时捕捉和预处理数据,包括采用Flink的transform算子对数据进行校验和处理,将处理后的数据sink到大数据的HDFS文件系统,交由下一步的大数据并行框架进行分析建模与训练,实现基于流计算和大数据平台的实时交通流预测。实验结果表明,Flink能够实时捕捉和预处理交通流数据,把数据准时无误送入分布式文件系统中,在此基础上借助大数据框架下的并行分析和建模优势,在实时性数据分析与预测方面取得了较好的效果。

安全大数据驱动的核电厂实时概率安全分析研究现状与发展前景

概率安全分析(PSA)是保障核电厂安全运行的关键方法之一,现有PSA方法多依靠有限的数据和专家经验开展静态的安全风险量化,难以精确反映核电厂运行阶段真实动态风险。核电厂运行过程中已实时收集并存储了大量可用来记录和描述系统安全状态的安全大数据,其信息提取和PSA融合需求迫切,本文对安全大数据在核电厂安全保障中的地位进行介绍,并分析总结安全大数据驱动的核电厂实时概率安全分析研究现状和发展前景。结果表明:安全大数据的应用有助于全面分析核电厂实时运行风险,将由安全大数据转换而来的安全信息与现有概率安全分析方法有机融合以实现准确可信的核电厂实时安全分析是未来重要方向。

基于Spark Streaming流回归的煤矿瓦斯浓度实时预测

为了实时分析瓦斯监测流数据并对瓦斯浓度进行准确预测以实现瓦斯灾害实时预警,以实时流数据处理框架Spark Streaming构建基于流回归的瓦斯浓度实时预测系统。系统采用分布式流处理技术,可使基于回归算法的瓦斯浓度预测模型更新周期达到秒级,提高了瓦斯浓度预测精度,满足流式大数据处理的实时性要求。实验表明:应用Spark Streaming流回归预测系统在采样周期为5s的瓦斯监测数据流上进行实时预测时,预测平均均方根误差随模型更新周期的缩短而减小,模型更新周期可达15s,且更新周期为45s时预测总均方根误差最小,既能保证预测精度,又能提高瓦斯灾害预警时效。

Spark Streaming动态资源分配策略

针对Spark Streaming作为混合大数据计算平台流处理组件时资源调整周期长和不能满足多应用多用户个性化需求的问题,提出了一种多应用下动态资源分配策略(DRAM)。该策略增加了应用全局变量来控制动态资源分配过程。首先,获取历史执行数据反馈和应用全局变量;然后,进行资源增减计算;最后,进行资源增减执行。实验结果表明,所提策略能够有效调整应用资源配额,且在稳定数据流和不稳定数据流两种情况下,其处理延时相比原Spark平台的Streaming策略和Core策略都有所降低;同时该策略也能够提高集群资源利用率。

基于Spark Streaming的实时数据采集分析系统设计

大数据量的实时数据分析系统,需要快速的处理和响应。为了实现实时数据分析,本文设计了基于Spark Streaming的实时数据采集分析系统,并对有状态计算操作的基数计算的精确计算方法和估算方法进行了比较分析。实验表明,基于HyperLogLog++的基数估算方法在处理时间和存储占用空间上有明显优势,而计算偏差基本可以忽略不计,更适于大数据的基数估算。

BDS-3卫星钟差改正数短期预报方法研究

针对RTS实时数据流产品在网络传输中存在的延迟以及数据中断等问题,文中结合BDS-3卫星钟差改正数的特点,分析实时数据流中BDS-3卫星钟差改正数的完整率和精度,提出一种基于一次差分和滑动时间窗口的残差修正钟差改正数短期预报模型。利用文中模型与一次多项式模型、基于一次差分的一次多项式模型和灰色模型进行BDS-3卫星钟差改正数预报精度的对比实验,结果表明,文中模型的预报精度最高,5 min、10 min、15 min和20 min的平均预报精度分别达到0.12 ns、0.14 ns、0.19 ns和0.20 ns,有效降低误差累积效应对预报精度的影响,对于RT-PPP的研究和应用具有重要的意义。

基于Spark计算的大数据终端潜在异常识别仿真

本文主要针对大数据终端在实时处理过程中潜在的异常情况进行识别和仿真研究。基于Spark计算框架,提出了一种新的方法监测和识别大数据终端中可能存在的异常行为。通过使用实时数据流分析和机器学习技术,及时发现并处理潜在的异常情况,提高数据处理效率和质量,并最终提升系统的可靠性和稳定性。

基于大数据的地铁供电系统运行状态预测与维护

为提升地铁供电系统运行状态的预测与维护效能,本文使用大数据分析地铁供电系统电流特征,精确识别不同站点和线路的峰谷情况。通过发现电流数据中的异常偏离点,实现对异常点数据的历史统计和实时提醒功能。同时增强了对设备管理的宏观把控。利用大数据收集每类电能质量指标参数及相关关键因素数据构建训练样本,通过支持向量机模型得到新建城轨线路的电能质量预测数据。为了评估系统可用性,采用序贯Monte Carlo法进行计算。结果显示,大数据方法预测准确性,达到92%。随着故障数量的增加,大数据方法的耗时仍能维持相对较低的水平,分别为2.2、3.5、5.1分钟,为地铁供电系统的整体性能提升提供了强有力的支持。

支持实时流计算应用的关键技术研究进展

信息系统在进行知识的挖掘和管理时,需要处理各种形式的数据,流数据便是其中之一.流数据具有数据规模大、产生速度快且蕴含的知识具有较强时效性等特点,因而发展支持实时处理应用的流计算技术对于信息系统的知识管理十分重要.流计算系统可以追溯到29世纪90年代,至今已经经历了长足的发展.然而,当前多样化的知识管理需求和新一代的硬件架构为流计算系统带来了全新的挑战和机遇,催生出了一系列流计算领域的技术研究.首先介绍流计算系统的基本需求以及发展脉络,再按照编程接口、执行计划、资源调度和故障容错4个层次分别分析流计算系统领域的相关技术;最后,展望流计算技术在未来可能的研究方向和发展趋势.

一种船舶历史活动热力图实时生成策略

在船舶信息大数据系统中,针对在客户端向服务器请求大批量历史数据并实时绘制热力图时,会出现耗时较长、热力图效果失真以及地图软件漫游时跟随性差或卡顿等问题,提出一种改进的船舶历史活动热力图实时生成策略。该策略在基于像素聚类思想的热力图生成方法基础上,采用基于非线性变换的图像增强算法和像素补点方法,对传统方法绘制的热力图所存在的显示失真问题进行校正和优化,并在此基础上,提出动态加载与绘制策略和多线程实时成图策略,来生成船舶历史活动的热力图。试验表明,所提出策略在热力图绘制的效率、成图效果以及地图软件的交互体验上都得到了提升。

基于Spark大数据平台与改进Adaboost算法的医院预分检系统研究

目的:设计基于Spark大数据平台与改进Adaboost算法的医院预分检系统,用于医院就诊患者诊前分流,加速患者就医流程。方法:基于Spark大数据平台实时采集初次进入医院就诊患者的基础数据,将区块链技术应用于数据采集、存储与传输全过程,通过改进Adaboost算法对数据进行分析,采用2011—2020年联勤保障部队第九四〇医院10年间门诊患者的就诊数据为数据集,对患者在院内就诊进行快速甄别并引导就诊。分析基于Spark大数据平台与改进Adaboost算法的医院预分检系统应用效果。结果:改进Adaboost算法设置自定义限制权重阈值为0.52时,算法准确率为95.56%,预检分诊准确率较传统Adaboost算法提高4.24%。患者平均候诊时间由采用预分检系统前的0.8 h缩短为0.5 h,患者平均就诊时间由6 min缩短为4.8 min。结论:基于大数据平台与改进Adaboost算法的医院预分检系统能够提前将医院就诊患者进行诊前分流,提高分检效率和分检准确率,缓解医院就诊压力。

轧机镰刀弯状态识别技术研究与应用

针对某钢厂4300 mm精轧机在实际运行过程中存在的问题,提出以镰刀弯质量问题为突破口的方法,展开轧机镰刀弯状态识别技术研究。首先从对轧机和轧制工艺等多源数据进行多维角度挖掘和提取,然后以三维数字可视化方法为辅,建立设备机理和数据相融合的全方位的钢板镰刀弯和轧制状态自动分析模型,对当前钢坯的镰刀弯状态进行实时预测。结果表明,该方法对于促进生产稳定运行,提高产品质量保证能力具有重要意义。

机械结构动态优化的创新方法与实际应用研究

本研究提出一种基于深度学习的机械结构动态优化方法,利用实时数据分析与自适应控制算法进行性能调优。该方法能即时监控并调整机械结构的多项参数,从而实现其性能的实时优化。通过一系列实验,证明了该方法能有效提高机械结构的耐用性和运行效率。这一创新方法有望为高效、可靠的工业生产提供有力支持。