Sentiment Drift Detection and Analysis in Real Time Twitter Data Streams

Handling sentiment drifts in real time twitter data streams are a challen-ging task while performing sentiment classifications,because of the changes that occur in the sentiments of twitter users,with respect to time.The growing volume of tweets with sentiment drifts has led to the need for devising an adaptive approach to detect and handle this drift in real time.This work proposes an adap-tive learning algorithm-based framework,Twitter Sentiment Drift Analysis-Bidir-ectional Encoder Representations from Transformers(TSDA-BERT),which introduces a sentiment drift measure to detect drifts and a domain impact score to adaptively retrain the classification model with domain relevant data in real time.The framework also works on static data by converting them to data streams using the Kafka tool.The experiments conducted on real time and simulated tweets of sports,health care andfinancial topics show that the proposed system is able to detect sentiment drifts and maintain the performance of the classification model,with accuracies of 91%,87%and 90%,respectively.Though the results have been provided only for a few topics,as a proof of concept,this framework can be applied to detect sentiment drifts and perform sentiment classification on real time data streams of any topic.

Two-Phase Rate Adaptation Strategy for Improving Real-Time Video QoE in Mobile Networks

With the popularity of smart handheld devices, mobile streaming video has multiplied the global network traffic in recent years. A huge concern of users' quality of experience(Qo E) has made rate adaptation methods very attractive. In this paper, we propose a two-phase rate adaptation strategy to improve users' real-time video Qo E. First, to measure and assess video Qo E, we provide a continuous Qo E prediction engine modeled by RNN recurrent neural network. Different from traditional Qo E models which consider the Qo E-aware factors separately or incompletely, our RNN-Qo E model accounts for three descriptive factors(video quality, rebuffering, and rate change) and reflects the impact of cognitive memory and recency. Besides, the video playing is separated into the initial startup phase and the steady playback phase, and we takes different optimization goals for each phase: the former aims at shortening the startup delay while the latter ameliorates the video quality and the rebufferings. Simulation results have shown that RNN-Qo E can follow the subjective Qo E quite well, and the proposed strategy can effectively reduce the occurrence of rebufferings caused by the mismatch between the requested video rates and the fluctuated throughput and attains standout performance on real-time Qo E compared with classical rate adaption methods.

Improvement Design for Distributed Real-Time Stream Processing Systems

In the era of Big Data, typical architecture of distributed real-time stream processing systems is the combination of Flume, Kafka, and Storm. As a kind of distributed message system, Kafka has the characteristics of horizontal scalability and high throughput, which is manly deployed in many areas in order to address the problem of speed mismatch between message producers and consumers. When using Kafka, we need to quickly receive data sent by producers. In addition, we need to send data to consumers quickly. Therefore, the performance of Kafka is of critical importance to the performance of the whole stream processing system. In this paper, we propose the improved design of real-time stream processing systems, and focus on improving the Kafka's data loading process.We use Kafka cat to transfer data from the source to Kafka topic directly, which can reduce the network transmission. We also utilize the memory file system to accelerate the process of data loading, which can address the bottleneck and performance problems caused by disk I/O. Extensive experiments are conducted to evaluate the performance, which show the superiority of our improved design.

A Real-Time TCP Stream Reassembly Mechanism in High-Speed Network

With the continual growth of the variety and complexity of network crime means, the traditional packet feature matching cannot detect all kinds of intrusion behaviors completely. It is urgent to reassemble network stream to perform packet processing at a semantic level above the network layer. This paper presents an efficient TCP stream reassembly mechanism for real-time processing of high-speed network traffic. By analyzing the characteristics of network stream in high-speed network and TCP connection establishment process, several polices for designing the reassembly mechanism are built. Then, the reassembly implementation is elaborated in accordance with the policies. Finally, the reassembly mechanism is compared with the traditional reassembly mechanism by the network traffic captured in a typical gigabit gateway. Experiment results illustrate that the reassembly mechanism is efficient and can satisfy the real-time property requirement of traffic analysis system in high-speed network.

Real-time Audio &Video Transmission System Based on Visible Light Communication

With the increasing popularity of solid sate lighting devices, Visible Light Communication (VLC) is globally recognized as an advanced and promising technology to realize short-range, high speed as well as large capacity wireless data transmission. In this paper, we propose a prototype of real-time audio and video broadcast system using inexpensive commercially available light emitting diode (LED) lamps. Experimental results show that real-time high quality audio and video with the maximum distance of 3 m can be achieved through proper layout of LED sources and improvement of concentration effects. Lighting model within room environment is designed and simulated which indicates close relationship between layout of light sources and distribution of illuminance.

Priority-Based Real-Time Stream Coding over Multi-Channel with Packet Erasures

Multi-channel can be used to provide higher transmission ability to the bandwidth-intensive and delay-sensitive real-time streams. However, traditional channel capacity theories and coding schemes are seldom designed for the real-time streams with strict delay constraint, especially in multi-channel context. This paper considers a real-time stream system, where real-time messages with different importance should be transmitted through several packet erasure channels, and be decoded by the receiver within a fixed delay. Based on window erasure channels and i.i.d.(identically and independently distributed) erasure channels, we derive the Multi-channel Real-time Stream Transmission(MRST) capacity models for Symmetric Real-time(SR) streams and Asymmetric Real-time(AR) streams respectively. Moreover, for window erasures, a Maximum Equilibrium Intra-session Code(MEIC) is presented for SR and AR streams, and is shown able to asymptotically achieve the theoretical MRST capacity. For i.i.d. erasures, we propose an Adaptive Maximum Equilibrium Intra-session Code(AMEIC), and then prove AMEIC can closely approach the MRST transmission capacity. Finally, the performances of the proposed codes are verified by simulations.

符合Real-time CORBA规范的传感器网络监控系统设计

传感器网络在解决底层数据源和I/O设备的异构性、处理连续查询的实时性、服务质量自适应等方面提出了许多新的挑战。提出一种基于实时CORBA的传感器网络数据流处理框架,说明了符合实时中间件规范的数据流处理系统设计的方法和特点,通过实验对该框架进行了性能测试,验证了其处理连续查询的实时性。

View Direction Adaptive 360 Degree Video Streaming System Based on Projected Area

360 video streaming services over the network are becoming popular. In particular, it is easy to experience 360 video through the already popular smartphone. However, due to the nature of 360 video, it is difficult to provide stable streaming service in general network environment because the size of data to send is larger than that of conventional video. Also, the real user's viewing area is very small compared to the sending amount. In this paper, we propose a system that can provide high quality 360 video streaming services to the users more efficiently in the cloud. In particular, we propose a streaming system focused on using a head mount display (HMD).

基于Spark Streaming的气象自动站实时流处理与存储系统

在当前大数据技术蓬勃发展的时代,人们对气象数据的实时处理、数据质量、数据存储及大规模查询等要求也越来越高。针对现有气象自动站数据业务落地环节多,任务处理耦合紧但系统部署分散等问题,文中基于Spark Streaming的流式计算框架,研究使用Flume解析收集自动站原始数据,在Spark Streaming中设计融入自动站数据质控算法,最终通过对分布式数据库存储的表设计,使气象自动站数据具备高效率、高质量、高可靠的应用服务能力。性能测试结果表明,基于Spark Streaming的气象自动站数据实时流处理与存储系统,数据从文件采集、解码、流处理至入库的全流程能够在秒级完成,TB级数据查询响应为毫秒级,加权查询为秒级,完全满足自动站数据业务应用需求,从而为进一步提高气象自动站数据质量与服务水平提供基础支撑。

智能行车记录仪图像去雾系统的FPGA设计

雾霾天气下,交通道路能见度低,导致所采集到的视频画面退化、图像信息模糊,同时考虑传统系统处理实时性不高等问题,基于ZYNQ平台设计了一种图像去雾系统,并应用于智能行车记录仪系统中.首先,针对传统暗通道去雾算法在天空区域存在失真等问题,提出了一种分割天空区域的策略来修正图像复原参数;然后,针对计算全局大气光值时,需对整幅图像的像素排序消耗大量资源的问题,利用现场可编程门阵列(FPGA)并行运算的优势,提出一种帧迭代方法优化求取大气光值,同时优化了引导滤波的硬件设计;最后,将双路高清多媒体接口(HDMI)资源中,一路作为视频输入,另一路作为视频处理输出,搭建实时交通图像视频处理试验平台.试验结果表明,系统针对雾霾天气下的交通视频具有较好的去雾效果,尤其是可以解决天空区域去雾的失真问题.在对分辨率为1280像素×720像素的交通视频去雾时,可以达到30帧/s的处理速度,满足实时性要求.

快速视频去雾改进算法的FPGA实现

内窥镜去雾算法在医疗领域具有广泛应用,为临床医生提供清晰、实时的图像。去雾技术虽然已经取得较大的进步,但去雾算法的复杂度较高,在内窥镜等复杂情况下硬件实现较为困难。为了在硬件上实现内窥镜实时去雾效果,对暗通道先验算法进行改进,降低硬件资源消耗和时间复杂度。该改进算法选择适合硬件的大气光照强度估计值、透射率补偿值以及采用流水线结构实现有雾图像的处理。采用Xilinx的ZYNQ7020实现该算法硬件电路,实时处理分辨率为640×480的视频图像,速度可达到260 fps,消耗LUT仅为1.28 K,寄存器619个单元。实验结果表明,相比于传统算法,改进算法具有处理速度快、功耗低、可移植性强的特点,满足内窥镜需要实时处理视频的要求。

实时输出视频的水下主动偏振成像

针对在水下主动偏振成像方法中,需要机械旋转相机前的检偏器获取正交偏振态图像,导致无法实时输出去后向散射视频的问题,提出了实时输出视频的水下主动偏振成像方法。首先,本方法将正交偏振切换位置从相机端更换到光源端,而相机端的检偏器偏振状态保持不变,并论证了此改变并不影响去后向散射的效果;然后,设计了阵列正交偏振光源,即光源由偏振态正交的两组LED灯珠交错排列组成,通过独立的驱动电路分别控制两组灯珠,输出偏振态正交的偏振光,从硬件上取消了偏振态切换的机械结构;第三,设计了同步控制装置,分别控制光源输出的偏振态和相机拍照,实现了正交偏振态图像数据的采集;最后,通过流水线式主动偏振图像处理方法,实现实时的视频输出。实验结果表明,本方法可以实时输出去后向散射视频,且在去后向散射性能上,与机械式主动偏振成像方法性能接近,两者去后向散射性能的背离度均低于4%,图像EME值增长倍数最大可达17.96。本方法具有结构简单、成本低、输出帧频不受影响以及相机选择性多等优点,能够广泛应用在水下成像领域。

基于Spark Streaming流回归的煤矿瓦斯浓度实时预测

为了实时分析瓦斯监测流数据并对瓦斯浓度进行准确预测以实现瓦斯灾害实时预警,以实时流数据处理框架Spark Streaming构建基于流回归的瓦斯浓度实时预测系统。系统采用分布式流处理技术,可使基于回归算法的瓦斯浓度预测模型更新周期达到秒级,提高了瓦斯浓度预测精度,满足流式大数据处理的实时性要求。实验表明:应用Spark Streaming流回归预测系统在采样周期为5s的瓦斯监测数据流上进行实时预测时,预测平均均方根误差随模型更新周期的缩短而减小,模型更新周期可达15s,且更新周期为45s时预测总均方根误差最小,既能保证预测精度,又能提高瓦斯灾害预警时效。

使用Spark Streaming的自适应实时DDoS检测和防御技术

分布式拒绝服务(distributed denial of service,DDoS)攻击是重要的安全威胁,网络速度的不断提高给传统的检测方法带来了新的挑战。以Spark等为代表的大数据处理技术,给网络安全的高速检测带来了新的契机。提出了一种基于Spark Streaming框架的自适应实时DDoS检测防御技术,通过对滑动窗口内源簇进行分组,并根据与各分组内源簇比例的偏差统计,检测出DDoS攻击流量。通过感知合法的网络流量,实现了对DDoS攻击的自适应快速检测和有效响应。实验结果表明,该技术可极大地提升检测能力,为保障网络服务性能和安全检测的可扩展性提供了一种可行的解决方案。

Spark Streaming动态资源分配策略

针对Spark Streaming作为混合大数据计算平台流处理组件时资源调整周期长和不能满足多应用多用户个性化需求的问题,提出了一种多应用下动态资源分配策略(DRAM)。该策略增加了应用全局变量来控制动态资源分配过程。首先,获取历史执行数据反馈和应用全局变量;然后,进行资源增减计算;最后,进行资源增减执行。实验结果表明,所提策略能够有效调整应用资源配额,且在稳定数据流和不稳定数据流两种情况下,其处理延时相比原Spark平台的Streaming策略和Core策略都有所降低;同时该策略也能够提高集群资源利用率。

基于GPU的北斗B1宽带复合信号实时发生器设计

为了实现北斗B1C+B1I信号的联合生成,提出一种基于软件无线电和图形处理器(graphics processing unit,GPU)加速的北斗B1宽带复合信号的实时生成方法,该方法针对单边带复数二进制偏移载波(single-sideband complex binary offset carrier,SCBOC)调制方式的信号体制进行设计,系统根据用户配置的接收机运动轨迹和星历文件,生成中频信号并通过射频端发射。为了进一步提升GPU并行运算速度,从优化设备内存结构、设计并行线程架构和统一计算设备架构流(compute unified device architecture stream,CUDA)加速3个方面,设计了基于异步运算的加速采样点数据计算的CUDA优化实现方案。测试结果证明,优化后的算法可以基于SCBOC调制实时生成北斗B1I+B1C信号,基于GTX3060的GPU平台,信号90 M采样率下能实现8颗卫星复合信号的实时生成。

基于流计算和大数据平台的实时交通流预测

目前交通流预测实时性差,很难满足在线分析和预测任务的需求,基于此提出一种Flink流计算框架和大数据平台结合的实时交通流预测方法。基于流计算框架实时捕捉和预处理数据,包括采用Flink的transform算子对数据进行校验和处理,将处理后的数据sink到大数据的HDFS文件系统,交由下一步的大数据并行框架进行分析建模与训练,实现基于流计算和大数据平台的实时交通流预测。实验结果表明,Flink能够实时捕捉和预处理交通流数据,把数据准时无误送入分布式文件系统中,在此基础上借助大数据框架下的并行分析和建模优势,在实时性数据分析与预测方面取得了较好的效果。

弹幕视频的实时互动性与会话机制研究

弹幕视频是以弹幕为在线评论手段的视频形式,具有“实时互动性”。该互动形式具有多模态、多线程特点,包含“人机”和“人际”两个互动空间,两个空间依靠会话角色转换进行连接。不同于线下会话,该互动过程以“我”的观看时间为参照建构共时心理时间系统,话语序列以并列式和序列式为主,且具有动态性。这种新的互动形式既是新媒体时代交际的必然需要,又是媒介对交际互动进行塑造和制约的结果。

基于Spark Streaming的实时数据采集分析系统设计

大数据量的实时数据分析系统,需要快速的处理和响应。为了实现实时数据分析,本文设计了基于Spark Streaming的实时数据采集分析系统,并对有状态计算操作的基数计算的精确计算方法和估算方法进行了比较分析。实验表明,基于HyperLogLog++的基数估算方法在处理时间和存储占用空间上有明显优势,而计算偏差基本可以忽略不计,更适于大数据的基数估算。

BDS-3卫星钟差改正数短期预报方法研究

针对RTS实时数据流产品在网络传输中存在的延迟以及数据中断等问题,文中结合BDS-3卫星钟差改正数的特点,分析实时数据流中BDS-3卫星钟差改正数的完整率和精度,提出一种基于一次差分和滑动时间窗口的残差修正钟差改正数短期预报模型。利用文中模型与一次多项式模型、基于一次差分的一次多项式模型和灰色模型进行BDS-3卫星钟差改正数预报精度的对比实验,结果表明,文中模型的预报精度最高,5 min、10 min、15 min和20 min的平均预报精度分别达到0.12 ns、0.14 ns、0.19 ns和0.20 ns,有效降低误差累积效应对预报精度的影响,对于RT-PPP的研究和应用具有重要的意义。