Priority-Based Real-Time Stream Coding over Multi-Channel with Packet Erasures

Multi-channel can be used to provide higher transmission ability to the bandwidth-intensive and delay-sensitive real-time streams. However, traditional channel capacity theories and coding schemes are seldom designed for the real-time streams with strict delay constraint, especially in multi-channel context. This paper considers a real-time stream system, where real-time messages with different importance should be transmitted through several packet erasure channels, and be decoded by the receiver within a fixed delay. Based on window erasure channels and i.i.d.(identically and independently distributed) erasure channels, we derive the Multi-channel Real-time Stream Transmission(MRST) capacity models for Symmetric Real-time(SR) streams and Asymmetric Real-time(AR) streams respectively. Moreover, for window erasures, a Maximum Equilibrium Intra-session Code(MEIC) is presented for SR and AR streams, and is shown able to asymptotically achieve the theoretical MRST capacity. For i.i.d. erasures, we propose an Adaptive Maximum Equilibrium Intra-session Code(AMEIC), and then prove AMEIC can closely approach the MRST transmission capacity. Finally, the performances of the proposed codes are verified by simulations.

Improvement Design for Distributed Real-Time Stream Processing Systems

In the era of Big Data, typical architecture of distributed real-time stream processing systems is the combination of Flume, Kafka, and Storm. As a kind of distributed message system, Kafka has the characteristics of horizontal scalability and high throughput, which is manly deployed in many areas in order to address the problem of speed mismatch between message producers and consumers. When using Kafka, we need to quickly receive data sent by producers. In addition, we need to send data to consumers quickly. Therefore, the performance of Kafka is of critical importance to the performance of the whole stream processing system. In this paper, we propose the improved design of real-time stream processing systems, and focus on improving the Kafka's data loading process.We use Kafka cat to transfer data from the source to Kafka topic directly, which can reduce the network transmission. We also utilize the memory file system to accelerate the process of data loading, which can address the bottleneck and performance problems caused by disk I/O. Extensive experiments are conducted to evaluate the performance, which show the superiority of our improved design.

Sentiment Drift Detection and Analysis in Real Time Twitter Data Streams

Handling sentiment drifts in real time twitter data streams are a challen-ging task while performing sentiment classifications,because of the changes that occur in the sentiments of twitter users,with respect to time.The growing volume of tweets with sentiment drifts has led to the need for devising an adaptive approach to detect and handle this drift in real time.This work proposes an adap-tive learning algorithm-based framework,Twitter Sentiment Drift Analysis-Bidir-ectional Encoder Representations from Transformers(TSDA-BERT),which introduces a sentiment drift measure to detect drifts and a domain impact score to adaptively retrain the classification model with domain relevant data in real time.The framework also works on static data by converting them to data streams using the Kafka tool.The experiments conducted on real time and simulated tweets of sports,health care andfinancial topics show that the proposed system is able to detect sentiment drifts and maintain the performance of the classification model,with accuracies of 91%,87%and 90%,respectively.Though the results have been provided only for a few topics,as a proof of concept,this framework can be applied to detect sentiment drifts and perform sentiment classification on real time data streams of any topic.

Theoretical analysis of fluid mixing time in liquid-continuous impinging streams reactor

The mixing time of impact zone in liquid-continuous impinging streams reactor(LISR) is theoretically calculated by empirical model and modern micromixing model of the fluid mixing process, and the variation laws of macromixing time and micromixing time are quantitatively discussed. The results show that under a continuous and stable operating condition, as the paddle speed increases, the macromixing time and micromixing time calculated by the two models both decrease, even in a linkage equilibrium state. Simultaneously, as the paddle speed increases, the results figured by the two models tend to be consistent. It indicates that two models both are more suitable for calculation of mixing time in high paddle speed. Compared with the existing experimental results of this type of reactor, the mixing time computed in the speed of 1500 r/min is closer to it. These conclusions can provide an important reference for systematically studying the strengthening mechanism of LISR under continuous mixing conditions.

A Real-Time TCP Stream Reassembly Mechanism in High-Speed Network

With the continual growth of the variety and complexity of network crime means, the traditional packet feature matching cannot detect all kinds of intrusion behaviors completely. It is urgent to reassemble network stream to perform packet processing at a semantic level above the network layer. This paper presents an efficient TCP stream reassembly mechanism for real-time processing of high-speed network traffic. By analyzing the characteristics of network stream in high-speed network and TCP connection establishment process, several polices for designing the reassembly mechanism are built. Then, the reassembly implementation is elaborated in accordance with the policies. Finally, the reassembly mechanism is compared with the traditional reassembly mechanism by the network traffic captured in a typical gigabit gateway. Experiment results illustrate that the reassembly mechanism is efficient and can satisfy the real-time property requirement of traffic analysis system in high-speed network.

基于Spark Streaming的气象自动站实时流处理与存储系统

在当前大数据技术蓬勃发展的时代,人们对气象数据的实时处理、数据质量、数据存储及大规模查询等要求也越来越高。针对现有气象自动站数据业务落地环节多,任务处理耦合紧但系统部署分散等问题,文中基于Spark Streaming的流式计算框架,研究使用Flume解析收集自动站原始数据,在Spark Streaming中设计融入自动站数据质控算法,最终通过对分布式数据库存储的表设计,使气象自动站数据具备高效率、高质量、高可靠的应用服务能力。性能测试结果表明,基于Spark Streaming的气象自动站数据实时流处理与存储系统,数据从文件采集、解码、流处理至入库的全流程能够在秒级完成,TB级数据查询响应为毫秒级,加权查询为秒级,完全满足自动站数据业务应用需求,从而为进一步提高气象自动站数据质量与服务水平提供基础支撑。

Solution and its application of transient stream/groundwater model subjected to time-dependent vertical seepage

Based on the first linearized Boussincsq equation, the analytical solution of the transient groundwater model, which is used for describing phreatic flow in a semiinfinite aquifer bounded by a linear stream and subjected to time-dependent vertical seepage, is derived out by Laplace transform and the convolution integral. According to the mathematical characteristics of the solution, different methods for estimating aquifer parameters are constructed to satisfy different hydrological conditions. Then, tile equation for estimating water exchange between stream and aquifer is proposed, and a recursion equation or estimating the intensity of phreatic evaporation is also proposed. A phreatic aquifer stream system located in Huaibei Plain, Anhui Province, China, is taken as an example to demonstrate tile estimation process of the methods stated herein.

运用价值流分析实现急诊救治零跑动的实践与探讨

目的通过价值流分析来识别急诊诊疗过程中医患双方人力资源的隐性浪费,并利用信息等管理手段来提高急救时间的利用率。方法分析急诊抢救区非绿通患者、家属及医疗工作者的诊疗,确定时间浪费的爆发点,构建以急救区域为中心的信息、物流系统,并对比优化前后的结果。结果优化后患者用药时间、CT时间、检验时间、流程时间均较优化前缩短,时间有效率高于优化前,差异均具有统计学意义(p<0.05);医护人员流程PT、LT均较优化前缩短,时间有效率高于优化前,差异均具有统计学意义(p<0.05);家属跑动次数从6次降为0次;医护接触次数较优化前下降,患者满意度较优化前升高,差异均具有统计学意义(p<0.05)。结论利用价值流分析并构建的以急救区域为中心的信息、物流系统,减少医患双方劳动力的隐性浪费,提高急救时间的利用率。

Spark Streaming动态资源分配策略

针对Spark Streaming作为混合大数据计算平台流处理组件时资源调整周期长和不能满足多应用多用户个性化需求的问题,提出了一种多应用下动态资源分配策略(DRAM)。该策略增加了应用全局变量来控制动态资源分配过程。首先,获取历史执行数据反馈和应用全局变量;然后,进行资源增减计算;最后,进行资源增减执行。实验结果表明,所提策略能够有效调整应用资源配额,且在稳定数据流和不稳定数据流两种情况下,其处理延时相比原Spark平台的Streaming策略和Core策略都有所降低;同时该策略也能够提高集群资源利用率。

使用Spark Streaming的自适应实时DDoS检测和防御技术

分布式拒绝服务(distributed denial of service,DDoS)攻击是重要的安全威胁,网络速度的不断提高给传统的检测方法带来了新的挑战。以Spark等为代表的大数据处理技术,给网络安全的高速检测带来了新的契机。提出了一种基于Spark Streaming框架的自适应实时DDoS检测防御技术,通过对滑动窗口内源簇进行分组,并根据与各分组内源簇比例的偏差统计,检测出DDoS攻击流量。通过感知合法的网络流量,实现了对DDoS攻击的自适应快速检测和有效响应。实验结果表明,该技术可极大地提升检测能力,为保障网络服务性能和安全检测的可扩展性提供了一种可行的解决方案。

BDS-3卫星钟差改正数短期预报方法研究

针对RTS实时数据流产品在网络传输中存在的延迟以及数据中断等问题,文中结合BDS-3卫星钟差改正数的特点,分析实时数据流中BDS-3卫星钟差改正数的完整率和精度,提出一种基于一次差分和滑动时间窗口的残差修正钟差改正数短期预报模型。利用文中模型与一次多项式模型、基于一次差分的一次多项式模型和灰色模型进行BDS-3卫星钟差改正数预报精度的对比实验,结果表明,文中模型的预报精度最高,5 min、10 min、15 min和20 min的平均预报精度分别达到0.12 ns、0.14 ns、0.19 ns和0.20 ns,有效降低误差累积效应对预报精度的影响,对于RT-PPP的研究和应用具有重要的意义。

一个安全的CORBA音/视频流服务模型:SecStream

通过对传统CORBA(common object request broker architecture)体系结构的扩展,OMG(Object Management Group)提出的CORBA音/视频流服务规范使得多媒体应用可以获得分布式对象计算在开放性、标准化等方面的优势.但该规范在安全方面有所欠缺,没有提供解决分布多媒体应用所面临的安全问题的完整方案.通过引入CORBA安全服务,并定义数据安全服务的概念,对原有流服务体系结构进行了安全扩展,提出了一个完善的安全流服务模型SecStream,对其进行设计和实现,并加以实践应用,验证了该模型可以灵活地提供多种安全保证,并具有较大的可扩展性.

基于Spark Streaming的实时数据采集分析系统设计

大数据量的实时数据分析系统,需要快速的处理和响应。为了实现实时数据分析,本文设计了基于Spark Streaming的实时数据采集分析系统,并对有状态计算操作的基数计算的精确计算方法和估算方法进行了比较分析。实验表明,基于HyperLogLog++的基数估算方法在处理时间和存储占用空间上有明显优势,而计算偏差基本可以忽略不计,更适于大数据的基数估算。

多台机器流水作业中存在调整时间的Lot-streaming问题

多台机器流水作业的 Lot-streaming问题（简称LS），以往的研究都不考虑调整时间，固定分批数，寻找最优分批大小；本文对机器引入调整时间，研究同时决定最优分批数及分批大小，并给出了相应最优算法．

一种自注意力序列模型的视频流长期预测方法

视频流量预测是实现传输带宽精准分配和提高互联网业务服务质量的关键技术。然而视频流量固有的高速率变异性、长期依赖性和短期依赖性使得其难以快速、精准、长期预测,具体表现为:①现有预测序列依赖关系的模型复杂度高;②预测模型失效快。针对视频流精准预测问题,提出了画面组帧结构特征嵌入的自注意力序列模型。自注意力序列模型对离散数据非线性关系的建模能力强,基于视频帧的特点和相关分析的发现,首次将时间序列自注意力模型应用于视频流量长期预测。现有时间序列自注意力模型无法对视频帧的类别特征有效表示,通过引入基于画面组帧结构嵌入层,将画面组帧结构信息有效嵌入时间序列,提升模型的准确度。结果表明,所提基于画面组帧结构特征嵌入的自注意力序列模型相比于现有的长短期记忆网络模型、卷积神经网络模型等,推理速度快,预测精度在平均绝对误差指标上至少降低约32%。

面向开集识别的稳健测试时适应方法

开集识别旨在研究测试阶段突现未见类别对于机器学习模型的挑战,以期学习模型既能分类已见类别又可识别/拒绝未见类别,是确保机器学习模型能够在开放世界中高效稳健部署的重要技术.既有开集识别技术通常假设已见类别的协变量分布在训练与测试阶段维持不变.然而在实际场景中,类别的协变量分布常不断变化.直接利用既有技术不再奏效,其性能甚至劣于基线方案.因此,亟需研究新型开集识别方法,使其能不断适应协变量分布偏移,以期模型在测试阶段既能稳健分类已见类别又可识别未见类别.将此新问题设置命名为开放世界适应问题(AOW),并提出了一种开放测试时适应方法(OTA).该方法基于无标注测试数据优化自适应熵损失与开集熵损失更新模型,维持对已见类的既有判别能力,同时增强了识别未见类的能力.大量实验分析表明,该方法在多组基准数据集、多组不同协变量偏移程度下均稳健地优于现有先进的开集识别方法.

支持实时流计算应用的关键技术研究进展

信息系统在进行知识的挖掘和管理时,需要处理各种形式的数据,流数据便是其中之一.流数据具有数据规模大、产生速度快且蕴含的知识具有较强时效性等特点,因而发展支持实时处理应用的流计算技术对于信息系统的知识管理十分重要.流计算系统可以追溯到29世纪90年代,至今已经经历了长足的发展.然而,当前多样化的知识管理需求和新一代的硬件架构为流计算系统带来了全新的挑战和机遇,催生出了一系列流计算领域的技术研究.首先介绍流计算系统的基本需求以及发展脉络,再按照编程接口、执行计划、资源调度和故障容错4个层次分别分析流计算系统领域的相关技术;最后,展望流计算技术在未来可能的研究方向和发展趋势.

基于Spark Streaming的实时日志处理平台设计与实现

本文基于Hadoop Yarn平台,通过Spark Streaming流数据处理技术实时处理Web日志、系统日志、监控日志等,很好地解决了目前存在的问题,并使用HBase(Hadoop No SQL Database)作为数据库存储,更好地适应日志数据规模的增长。

基于自适应流聚类的多站时差协同信号分选方法

多站时差协同信号分选是解决脉间波形捷变雷达信号分选的有效方法。已有批处理多站时差协同信号分选方法实时性不足,且存在无法有效挖掘雷达脉冲流动态变化特性的问题,该文提出一种基于自适应流聚类的多站时差协同信号分选方法。针对已有流聚类算法在雷达信号样本密度不平衡的情况下存在容易增批分选的问题,提出利用多站时差测量误差估计量对流聚类生成簇进行自适应检测与合并,有效抑制雷达簇增批现象的发生,以解决分选增批的问题。同时,通过利用多站时差测量误差估计量实现在线聚类阈值自适应设置,提高脉冲归属正确雷达簇的概率,减少在线聚类算法迭代次数。使用基于时间衰减窗口模型的未知簇检测,提高聚类概要的更新速度,避免了有效样本数过少而导致的聚类失败,提高了分选算法的环境适应性。仿真结果表明,与现有分选算法相比,面对不平衡复杂脉间波形捷变信号,该算法能够进行抑制增批,有效分选出正确雷达结果;面对多部雷达的出现/失活/复活演化过程,该算法可精准检测并展示动态演化特性;面对多部复杂脉间波形捷变信号,对比已有算法的算法复杂度大幅降低,特别在脉冲样本数密集情况下,能更好的保证分选处理的实时性;面对较大的多站时差测量误差和脉冲丢失率情况下,具有较高的分选正确率。

基于GPU的北斗B1宽带复合信号实时发生器设计

为了实现北斗B1C+B1I信号的联合生成,提出一种基于软件无线电和图形处理器(graphics processing unit,GPU)加速的北斗B1宽带复合信号的实时生成方法,该方法针对单边带复数二进制偏移载波(single-sideband complex binary offset carrier,SCBOC)调制方式的信号体制进行设计,系统根据用户配置的接收机运动轨迹和星历文件,生成中频信号并通过射频端发射。为了进一步提升GPU并行运算速度,从优化设备内存结构、设计并行线程架构和统一计算设备架构流(compute unified device architecture stream,CUDA)加速3个方面,设计了基于异步运算的加速采样点数据计算的CUDA优化实现方案。测试结果证明,优化后的算法可以基于SCBOC调制实时生成北斗B1I+B1C信号,基于GTX3060的GPU平台,信号90 M采样率下能实现8颗卫星复合信号的实时生成。