Improvement Design for Distributed Real-Time Stream Processing Systems

In the era of Big Data, typical architecture of distributed real-time stream processing systems is the combination of Flume, Kafka, and Storm. As a kind of distributed message system, Kafka has the characteristics of horizontal scalability and high throughput, which is manly deployed in many areas in order to address the problem of speed mismatch between message producers and consumers. When using Kafka, we need to quickly receive data sent by producers. In addition, we need to send data to consumers quickly. Therefore, the performance of Kafka is of critical importance to the performance of the whole stream processing system. In this paper, we propose the improved design of real-time stream processing systems, and focus on improving the Kafka's data loading process.We use Kafka cat to transfer data from the source to Kafka topic directly, which can reduce the network transmission. We also utilize the memory file system to accelerate the process of data loading, which can address the bottleneck and performance problems caused by disk I/O. Extensive experiments are conducted to evaluate the performance, which show the superiority of our improved design.

A Real-Time TCP Stream Reassembly Mechanism in High-Speed Network

With the continual growth of the variety and complexity of network crime means, the traditional packet feature matching cannot detect all kinds of intrusion behaviors completely. It is urgent to reassemble network stream to perform packet processing at a semantic level above the network layer. This paper presents an efficient TCP stream reassembly mechanism for real-time processing of high-speed network traffic. By analyzing the characteristics of network stream in high-speed network and TCP connection establishment process, several polices for designing the reassembly mechanism are built. Then, the reassembly implementation is elaborated in accordance with the policies. Finally, the reassembly mechanism is compared with the traditional reassembly mechanism by the network traffic captured in a typical gigabit gateway. Experiment results illustrate that the reassembly mechanism is efficient and can satisfy the real-time property requirement of traffic analysis system in high-speed network.

Stream-computing of High Accuracy On-board Real-time Cloud Detection for High Resolution Optical Satellite Imagery

This paper focuses on the time efficiency for machine vision and intelligent photogrammetry, especially high accuracy on-board real-time cloud detection method. With the development of technology, the data acquisition ability is growing continuously and the volume of raw data is increasing explosively. Meanwhile, because of the higher requirement of data accuracy, the computation load is also becoming heavier. This situation makes time efficiency extremely important. Moreover, the cloud cover rate of optical satellite imagery is up to approximately 50%, which is seriously restricting the applications of on-board intelligent photogrammetry services. To meet the on-board cloud detection requirements and offer valid input data to subsequent processing, this paper presents a stream-computing of high accuracy on-board real-time cloud detection solution which follows the “bottom-up” understanding strategy of machine vision and uses multiple embedded GPU with significant potential to be applied on-board. Without external memory, the data parallel pipeline system based on multiple processing modules of this solution could afford the “stream-in, processing, stream-out” real-time stream computing. In experiments, images of GF-2 satellite are used to validate the accuracy and performance of this approach, and the experimental results show that this solution could not only bring up cloud detection accuracy, but also match the on-board real-time processing requirements.

Priority-Based Real-Time Stream Coding over Multi-Channel with Packet Erasures

Multi-channel can be used to provide higher transmission ability to the bandwidth-intensive and delay-sensitive real-time streams. However, traditional channel capacity theories and coding schemes are seldom designed for the real-time streams with strict delay constraint, especially in multi-channel context. This paper considers a real-time stream system, where real-time messages with different importance should be transmitted through several packet erasure channels, and be decoded by the receiver within a fixed delay. Based on window erasure channels and i.i.d.(identically and independently distributed) erasure channels, we derive the Multi-channel Real-time Stream Transmission(MRST) capacity models for Symmetric Real-time(SR) streams and Asymmetric Real-time(AR) streams respectively. Moreover, for window erasures, a Maximum Equilibrium Intra-session Code(MEIC) is presented for SR and AR streams, and is shown able to asymptotically achieve the theoretical MRST capacity. For i.i.d. erasures, we propose an Adaptive Maximum Equilibrium Intra-session Code(AMEIC), and then prove AMEIC can closely approach the MRST transmission capacity. Finally, the performances of the proposed codes are verified by simulations.

CeTrivium:A Stream Cipher Based on Cellular Automata for Securing Real-Time Multimedia Transmission

Due to their significant correlation and redundancy,conventional block cipher cryptosystems are not efficient in encryptingmultimedia data.Streamciphers based onCellularAutomata(CA)can provide amore effective solution.The CA have recently gained recognition as a robust cryptographic primitive,being used as pseudorandom number generators in hash functions,block ciphers and stream ciphers.CA have the ability to perform parallel transformations,resulting in high throughput performance.Additionally,they exhibit a natural tendency to resist fault attacks.Few stream cipher schemes based on CA have been proposed in the literature.Though,their encryption/decryption throughput is relatively low,which makes them unsuitable formultimedia communication.Trivium and Grain are efficient stream ciphers that were selected as finalists in the eSTREAM project,but they have proven to be vulnerable to differential fault attacks.This work introduces a novel and scalable stream cipher named CeTrivium,whose design is based on CA.CeTrivium is a 5-neighborhood CA-based streamcipher inspired by the designs of Trivium and Grain.It is constructed using three building blocks:the Trivium(Tr)block,the Nonlinear-CA(NCA)block,and the Nonlinear Mixing(NM)block.The NCA block is a 64-bit nonlinear hybrid 5-neighborhood CA,while the Tr block has the same structure as the Trivium stream cipher.The NM block is a nonlinear,balanced,and reversible Boolean function that mixes the outputs of the Tr and NCA blocks to produce a keystream.Cryptanalysis of CeTrivium has indicated that it can resist various attacks,including correlation,algebraic,fault,cube,Meier and Staffelbach,and side channel attacks.Moreover,the scheme is evaluated using histogramand spectrogramanalysis,aswell as several differentmeasurements,including the correlation coefficient,number of samples change rate,signal-to-noise ratio,entropy,and peak signal-to-noise ratio.The performance of CeTrivium is evaluated and compared with other state-of-the-art techniques.CeTrivium outperforms them in terms of encryption throughput while maintaining high security.CeTrivium has high encryption and decryption speeds,is scalable,and resists various attacks,making it suitable for multimedia communication.

Partitioned k-Exclusion Real-Time Locking Protocol Motivated by Multicore Multi-GPU Systems

Graphic processing units （GPUs） have been widely recognized as cost-efficient co-processors with acceptable size, weight, and power consumption. However, adopting GPUs in real-time systems is still challenging, due to the lack in framework for real-time analysis. In order to guarantee real-time requirements while maintaining system utilization ~in modern heterogeneous systems, such as multicore multi-GPU systems, a novel suspension-based k-exclusion real-time locking protocol and the associated suspension-aware schedulability analysis are proposed. The proposed protocol provides a synchronization framework that enables multiple GPUs to be efficiently integrated in multicore real-time systems. Comparative evaluations show that the proposed methods improve upon the existing work in terms of schedulability.

Real-Time Communication Method for mHealth Base on Extended XMPP Protocol

Considering characteristic of mHealth communication and problems of existing methods, this paper presents a real-time communication method for mHealth based on extended XMPP protocol. The method can maintain the role status efficiently and reduce data latency during the communication process. Meanwhile, it can be extended flexibly to meet increasing communication demands of mHealth services. Furthermore, a system framework is presented to support telemonitoring scene. Finally, system implementation and feasibility tests verify the effectiveness of the method and framework.

Contention-Based Beaconless Real-Time Routing Protocol for Wireless Sensor Networks

This paper presents a novel real-time routing protocol, called CBRR, with less energy consumption for wireless sensor networks (WSNs). End-to-End real-time requirements are fulfilled with speed or delay constraint at each hop through integrating the contention and neighbor table mechanisms. More precisely, CBRR maintains a neighbor table via the contention mechanism being dependent on wireless broadcast instead of beacons. Comprehensive simulations show that CBRR can not only achieve higher performance in static networks, but also work well for dynamic networks.

Implementation of Mobile Streaming Media Player Based on BREW

Nowadays mobile streaming service through cell phone is becoming the highlight of new value-added mobile services. Based on the present CDMAlx wireless data network and Binary Runtime Environment for Wireless （BREW） platform, adopting compression technologies of H.264 and QCP, a set of streaming media players are designed and implemented, and the principle, structure, key technologies and performance analysis of this system are introduced. This player works well in practice.

Two-Phase Rate Adaptation Strategy for Improving Real-Time Video QoE in Mobile Networks

With the popularity of smart handheld devices, mobile streaming video has multiplied the global network traffic in recent years. A huge concern of users' quality of experience(Qo E) has made rate adaptation methods very attractive. In this paper, we propose a two-phase rate adaptation strategy to improve users' real-time video Qo E. First, to measure and assess video Qo E, we provide a continuous Qo E prediction engine modeled by RNN recurrent neural network. Different from traditional Qo E models which consider the Qo E-aware factors separately or incompletely, our RNN-Qo E model accounts for three descriptive factors(video quality, rebuffering, and rate change) and reflects the impact of cognitive memory and recency. Besides, the video playing is separated into the initial startup phase and the steady playback phase, and we takes different optimization goals for each phase: the former aims at shortening the startup delay while the latter ameliorates the video quality and the rebufferings. Simulation results have shown that RNN-Qo E can follow the subjective Qo E quite well, and the proposed strategy can effectively reduce the occurrence of rebufferings caused by the mismatch between the requested video rates and the fluctuated throughput and attains standout performance on real-time Qo E compared with classical rate adaption methods.

Design and Implementation of a Proxy Caching System for Streaming Media

With the widespread use of streaming media application on the Internet, a significant change in Internet workload will be provoked. Caching is one of the applied techniques for enhancing the scalability of streaming system and reducing the workload of server/network. Aiming at the characteristics of broadband network in community, we propose a popularity-based server-proxy caching strategy for streaming medias, and implement the prototype of streaming proxy caching based on this strategy, using RTSP as control protocol and RTP for content transport. This system can play a role in decreasing server load, reducing the traffic from streaming server to proxy, and improving the start-up latency of the client. Key words streaming server - proxy - cache - streaming media - real time streaming protocol CLC number TP 302 - TP 333 Foundation item: Supported by the National High Technology Development 863 Program of China (2001AA111011).Biography: Tan Jin (1962-), male, Ph. D candidate, research direction: network communications, multimedia technologies, and web caching.

Transmission Considerations with QoS Support to Deliver Real-Time Distributed Speech Recognition Applications

Distributed speech recognition (DSR) applications have certain QoS (Quality of service) requirements in terms of latency, packet loss rate, etc. To deliver quality guaranteed DSR application over wirelined or wireless links, some QoS mechanisms should be provided. We put forward a RTP/RSVP transmission scheme with DSR-specific payload and QoS parameters by modifying the present WAP protocol stack. The simulation result shows that this scheme will provide adequate network bandwidth to keep the real-time transport of DSR data over either wirelined or wireless channels.

RESEARCH ON GROUPING STRATEGY OF SIP-BASED STREAMING MEDIA P2P LIVE BROADCAST NETWORK

The rapid development of Internet has led to the explosion of information sharing, and how to supervise the sharing is a main research topic on current Internet. Aiming at the disadvantage that the current Peer-to-Peer (P2P) is hard to manage and control, this paper presents a Session Initial Protocol (SIP)-based P2P network of three-level architecture. SIP middleware is introduced to the middle level of the three-layer architecture. By the connection function of the SIP signaling, the P2P transmission on media-level can be controlled. Using SIP’s register and authentication function, the manage layer can manage the whole P2P network. Based on the aforementioned architecture, this paper investigates the grouping strategy on a live broadcast application in P2P network. Combined with the function of SIP register, the paper works on several grouping strategies, sets up models to manage users by grouping them, presents a weight-based K-means IP address grouping algorithm, and realizes it. The experiment shows that the grouping strategy presented in this paper can solve the problem of group sharing of network resource, and can realize the efficient-sharing, reasonable-distributing of network resource.

View Direction Adaptive 360 Degree Video Streaming System Based on Projected Area

360 video streaming services over the network are becoming popular. In particular, it is easy to experience 360 video through the already popular smartphone. However, due to the nature of 360 video, it is difficult to provide stable streaming service in general network environment because the size of data to send is larger than that of conventional video. Also, the real user's viewing area is very small compared to the sending amount. In this paper, we propose a system that can provide high quality 360 video streaming services to the users more efficiently in the cloud. In particular, we propose a streaming system focused on using a head mount display (HMD).

A Stream Control Transmission Protocol Based OAM System of 3G Cellular Network

OAM (Operations, Administration and Maintenance) system is a very impo rtant component of 3G cellular network. In order to acquire overall managemen t, fast response and steady operation, an SCTP (Stream Control Transmission Prot ocol) based OAM, i.e., SOAM system was proposed. SOAM implements new characters of SCTP such as multi-stream, enforced SACK and heartbeat mechanism on its tran sport layer. These characters help SOAM decrease the message transmission delay and accelerate the link failure detection. Besides, a new component named SOAM agent was introduced to improve the operation efficiency of SOAM. The experim ental results prove the proposed SOAM system achieves better performance on sign aling transmission compared with conventional TCP based OAM system.

Adaptive Learning Video Streaming with QoE in Multi-Home Heterogeneous Networks

In recent years,real-time video streaming has grown in popularity.The growing popularity of the Internet of Things(IoT)and other wireless heterogeneous networks mandates that network resources be carefully apportioned among versatile users in order to achieve the best Quality of Experience(QoE)and performance objectives.Most researchers focused on Forward Error Correction(FEC)techniques when attempting to strike a balance between QoE and performance.However,as network capacity increases,the performance degrades,impacting the live visual experience.Recently,Deep Learning(DL)algorithms have been successfully integrated with FEC to stream videos across multiple heterogeneous networks.But these algorithms need to be changed to make the experience better without sacrificing packet loss and delay time.To address the previous challenge,this paper proposes a novel intelligent algorithm that streams video in multi-home heterogeneous networks based on network-centric characteristics.The proposed framework contains modules such as Intelligent Content Extraction Module(ICEM),Channel Status Monitor(CSM),and Adaptive FEC(AFEC).This framework adopts the Cognitive Learning-based Scheduling(CLS)Module,which works on the deep Reinforced Gated Recurrent Networks(RGRN)principle and embeds them along with the FEC to achieve better performances.The complete framework was developed using the Objective Modular Network Testbed in C++(OMNET++),Internet networking(INET),and Python 3.10,with Keras as the front end and Tensorflow 2.10 as the back end.With extensive experimentation,the proposed model outperforms the other existing intelligentmodels in terms of improving the QoE,minimizing the End-to-End Delay(EED),and maintaining the highest accuracy(98%)and a lower Root Mean Square Error(RMSE)value of 0.001.

An ALM Model for Streaming Media Service in P2P-SIP Network

Application layer multicast （ALM） has been widely applied in Internet, as a substitute for IP multicast. However, it causes network load to increase because it adopts unlcast in data transmission, which limits the application. In this article, in order to improve the ALM performance in P2P-SIP network, an ALM model was proposed which reduced network load via merging messages in concast mode. Finally network simulations prove that this model has better benefit on single node load and average network load. Therefore, it is suitable for streaming media service in P2P-SIP Network.

基于STREAMS的X．25分组交换网访问软件的设计与实现

随着远程通信需求的不断增长和我国公用分组交换数据网的扩容，实现微机ＵＮＩＸ环境下的分组交换网访问协议软件，使我国成千上万的微机用户实现远程通信已迫在眉睫。本文在简单介绍了基于ＳＴＲＥＡＭＳ机制的通信软件开发的一般思路之后，着重叙述了基于ＳＴＲＥＡＭＳ机制的Ｘ．２５分组交换网协议访问软件的设计与实现．并对基于ＳＴＲＥＡＭＳ机制的通信软件开发方法作了简单评价．

一个安全的CORBA音/视频流服务模型:SecStream

通过对传统CORBA(common object request broker architecture)体系结构的扩展,OMG(Object Management Group)提出的CORBA音/视频流服务规范使得多媒体应用可以获得分布式对象计算在开放性、标准化等方面的优势.但该规范在安全方面有所欠缺,没有提供解决分布多媒体应用所面临的安全问题的完整方案.通过引入CORBA安全服务,并定义数据安全服务的概念,对原有流服务体系结构进行了安全扩展,提出了一个完善的安全流服务模型SecStream,对其进行设计和实现,并加以实践应用,验证了该模型可以灵活地提供多种安全保证,并具有较大的可扩展性.

基于STREAMS机制的IP/X.25软件的设计与实现

ＳＴＲＥＡＭＳ机制是ＵＮＩＸ系统为开发通信服务而提供的一套通用、灵活的开发工具，ＩＰ／Ｘ．２５软件则是ＴＣＰ／ＩＰ网络与Ｘ．２５公用数据网互连以及通过Ｘ．２５公用数据网实现ＴＣＰ／ＩＰ网络之间的互连的重要途径．文中在介绍基于ＳＴＲＥＡＭＳ机制的通信软件开发的一般思路后，着重叙述了ＩＰ／Ｘ．２５软件的功能和结构以及基于ＳＴＲＥＡＭＳ机制的设计与实现，并对基于ＳＴＲＥＡＭＳ机制的通信软件开发方法作了简单评价．