Two-Phase Rate Adaptation Strategy for Improving Real-Time Video QoE in Mobile Networks

With the popularity of smart handheld devices, mobile streaming video has multiplied the global network traffic in recent years. A huge concern of users' quality of experience(Qo E) has made rate adaptation methods very attractive. In this paper, we propose a two-phase rate adaptation strategy to improve users' real-time video Qo E. First, to measure and assess video Qo E, we provide a continuous Qo E prediction engine modeled by RNN recurrent neural network. Different from traditional Qo E models which consider the Qo E-aware factors separately or incompletely, our RNN-Qo E model accounts for three descriptive factors(video quality, rebuffering, and rate change) and reflects the impact of cognitive memory and recency. Besides, the video playing is separated into the initial startup phase and the steady playback phase, and we takes different optimization goals for each phase: the former aims at shortening the startup delay while the latter ameliorates the video quality and the rebufferings. Simulation results have shown that RNN-Qo E can follow the subjective Qo E quite well, and the proposed strategy can effectively reduce the occurrence of rebufferings caused by the mismatch between the requested video rates and the fluctuated throughput and attains standout performance on real-time Qo E compared with classical rate adaption methods.

A survey on image and video stitching

Image/video stitching is a technology for solving the field of view(FOV)limitation of images/videos.It stitches multiple overlapping images/videos to generate a wide-FOV image/video,and has been used in various fields such as sports broadcasting,video surveillance,street view,and entertainment.This survey reviews image/video stitching algorithms,with a particular focus on those developed in recent years.Image stitching first calculates the corresponding relationships between multiple overlapping images,deforms and aligns the matched images,and then blends the aligned images to generate a wide-FOV image.A seamless method is always adopted to eliminate such potential flaws as ghosting and blurring caused by parallax or objects moving across the overlapping regions.Video stitching is the further extension of image stitching.It usually stitches selected frames of original videos to generate a stitching template by performing image stitching algorithms,and the subsequent frames can then be stitched according to the template.Video stitching is more complicated with moving objects or violent camera movement,because these factors introduce jitter,shakiness,ghosting,and blurring.Foreground detection technique is usually combined into stitching to eliminate ghosting and blurring,while video stabilization algorithms are adopted to solve the jitter and shakiness.This paper further discusses panoramic stitching as a special-extension of image/video stitching.Panoramic stitching is currently the most widely used application in stitching.This survey reviews the latest image/video stitching methods,and introduces the fundamental principles/advantages/weaknesses of image/video stitching algorithms.Image/video stitching faces long-term challenges such as wide baseline,large parallax,and low-texture problem in the overlapping region.New technologies may present new opportunities to address these issues,such as deep learning-based semantic correspondence,and 3D image stitching.Finally,this survey discusses the challenges of image/video stitching and proposes potential solutions.

Real-time Audio &Video Transmission System Based on Visible Light Communication

With the increasing popularity of solid sate lighting devices, Visible Light Communication (VLC) is globally recognized as an advanced and promising technology to realize short-range, high speed as well as large capacity wireless data transmission. In this paper, we propose a prototype of real-time audio and video broadcast system using inexpensive commercially available light emitting diode (LED) lamps. Experimental results show that real-time high quality audio and video with the maximum distance of 3 m can be achieved through proper layout of LED sources and improvement of concentration effects. Lighting model within room environment is designed and simulated which indicates close relationship between layout of light sources and distribution of illuminance.

Adaptive Learning Video Streaming with QoE in Multi-Home Heterogeneous Networks

In recent years,real-time video streaming has grown in popularity.The growing popularity of the Internet of Things(IoT)and other wireless heterogeneous networks mandates that network resources be carefully apportioned among versatile users in order to achieve the best Quality of Experience(QoE)and performance objectives.Most researchers focused on Forward Error Correction(FEC)techniques when attempting to strike a balance between QoE and performance.However,as network capacity increases,the performance degrades,impacting the live visual experience.Recently,Deep Learning(DL)algorithms have been successfully integrated with FEC to stream videos across multiple heterogeneous networks.But these algorithms need to be changed to make the experience better without sacrificing packet loss and delay time.To address the previous challenge,this paper proposes a novel intelligent algorithm that streams video in multi-home heterogeneous networks based on network-centric characteristics.The proposed framework contains modules such as Intelligent Content Extraction Module(ICEM),Channel Status Monitor(CSM),and Adaptive FEC(AFEC).This framework adopts the Cognitive Learning-based Scheduling(CLS)Module,which works on the deep Reinforced Gated Recurrent Networks(RGRN)principle and embeds them along with the FEC to achieve better performances.The complete framework was developed using the Objective Modular Network Testbed in C++(OMNET++),Internet networking(INET),and Python 3.10,with Keras as the front end and Tensorflow 2.10 as the back end.With extensive experimentation,the proposed model outperforms the other existing intelligentmodels in terms of improving the QoE,minimizing the End-to-End Delay(EED),and maintaining the highest accuracy(98%)and a lower Root Mean Square Error(RMSE)value of 0.001.

基于国产FPGA的多源视频信号拼接系统设计

针对目前视频拼接技术存在动态视频数据速度慢、分辨率较低的问题,设计了一种能够将两路摄像头和两路高清晰度多媒体接口(high definition multimedia interface,HDMI)输入的动态视频数据进行拼接处理的系统。系统以紫光同创公司Pango logos系列现场可编程门阵列(field programmable gate array,FPGA)器件为核心,由时钟控制、寄存器配置、数据采集、先进先出(first in first out,FIFO)数据存储控制、第3代双倍速率同步动态随机存储(double-data-rate 3 synchronous dynamic RAM,DDR3 SDRAM)等逻辑功能模块组成。设计实现了DDR3 IP核的读写控制、图像缩放及视频拼接算法,使用Inserter和Debugger对主要功能模块进行了仿真测试,并进行了视频采集实验,结果表明,系统能够完成4路动态视频的拼接显示,分辨率达到1920×1080,刷新速率为60 fps,占用的查找表(look up table,LUT)资源为18%。支持多种动态数据输入方式,具有便携性好、帧率分辨率高、片上资源占用率低等特点,可用于动态视频拼接的应用场景。

Wireless ad hoc video transmission:a Bayesian network-based scheme

A novel bandwidth prediction and control scheme is proposed for video transmission over an ad boc network. The scheme is based on cross-layer, feedback, and Bayesian network techniques. The impacts of video quality are formulized and deduced. The relevant factors are obtained by a cross-layer mechanism or Feedback method. According to these relevant factors, the variable set and the Bayesian network topology are determined. Then a Bayesian network prediction model is constructed. The results of the prediction can be used as the bandwidth of the mobile ad hoc network （MANET）. According to the bandwidth, the video encoder is controlled to dynamically adjust and encode the right bit rates of a real-time video stream. Integrated simulation of a video streaming communication system is implemented to validate the proposed solution. In contrast to the conventional transfer scheme, the results of the experiment indicate that the proposed scheme can make the best use of the network bandwidth; there are considerable improvements in the packet loss and the visual quality of real-time video.K

三维仿真与虚实融合技术在视频监控系统中的应用

随着科技的不断发展,传统的视频监控正迎来新的突破。如今,监控相机在公共安全、交通安保、智慧城市以及日常生活等方面的应用越来越广泛,其中,基于虚实融合的视频可视化技术更是引领着这一变革的潮流。这种技术不仅优化运营管理,还增强了重点区域安全、应急效率,为管理者提供更准确的数据支持。本文主要研究基于虚实融合技术视频平台的技术方案,并对系统架构和主要功能进行了探讨。

基于无人机视频流的实时拼接方法研究

针对无人机视频流的实时拼接方法进行深入研究,提出一种多线程并行处理的无人机视频实时拼接方法,主要解决无人机视频流实时拼接问题。该方法充分利用计算机并行运算的特点,将视频实时拼接工作分解成多线程工作模式,主要由图像配准线程、非线性优化线程、闭环检测线程、图像拼接线程组成,其中图像配准线程实现对视频关键帧的自适应提取和配准,包括特征点检测、特征点匹配、图像粗配准;非线性优化线程完成对拼接参数模型的动态优化以最小化配准误差,及时修正实时配准引入的误差;闭环检测线程检测当前帧与历史帧存在的闭环图像对,通过闭环图像对进行非线性全局优化以最小化全局配准误差;图像拼接线程则根据优化后的配准参数动态生成拼接影像。通过4组实验数据验证了该方法的有效性和实用性,能有效消除多帧拼接的累积误差,实现对无人机视频流的实时拼接。

数字孪生收费站关键技术研究

由于高速公路收费站路面情况复杂,通行车辆作为数字孪生收费站中的核心动态实体,具有数量类型多、轨迹变化率高、交互需求多等特点,而当前以视频识别为核心的数字孪生系统技术架构并不适用于收费站场景。因此,文章针对收费站的特殊场景需求,对数字孪生收费站的核心技术及其融合架构进行研究,通过对模型渲染、时空融合、感知融合、虚实联动等方面的技术优化方向进行研究,为建立精准的数字孪生收费站系统提供方向。

基于图像拼接技术的动漫视频生成方法

针对在动漫图像拼接过程中出现的运算量大、速度慢、特征点提取精度低、维度高以及图像拼接效果差等问题,研究基于图像拼接技术的动漫视频生成方法。通过均值滤波预处理动漫图像,去除图像中的噪声和干扰后,采用加速稳健特征(SURF)算法提取滤波后动漫图像的特征点,经边缘相似匹配法完成图像特征点匹配后,利用加权平滑算法融合匹配后的图像,获得全景图,并将其导入Maya软件生成动漫视频。试验表明:该方法提取特征点精度高、维度低、运算量小、速度快,图像拼接效果清晰、连接紧密、过渡自然、没有色差,生成的动漫视频可以达到更加精美逼真的视觉效果。

RTP payload format for H.264/SVC scalable video coding

The scalable extension of H.264/AVC, known as scalable video coding or SVC, is currently the main focus of the Joint Video Team’s work. In its present working draft, the higher level syntax of SVC follows the design principles of H.264/AVC. Self-contained network abstraction layer units (NAL units) form natural entities for packetization. The SVC specification is by no means finalized yet, but nevertheless the work towards an optimized RTP payload format has already started. RFC 3984, the RTP payload specification for H.264/AVC has been taken as a starting point, but it became quickly clear that the scalable features of SVC require adaptation in at least the areas of capability/operation point signaling and documentation of the extended NAL unit header. This paper first gives an overview of the history of scalable video coding, and then reviews the video coding layer (VCL) and NAL of the latest SVC draft specification. Finally, it discusses different aspects of the draft SVC RTP payload format, in- cluding the design criteria, use cases, signaling and payload structure.

Design and implementation of multi-hop video transmission experiment system in VANET

Abts ract A wireless mutl i-hop videot ransmission experiment system is designed and implemented for vehiculra ad-hoc networks VANET and the rt ansm ission control protocol and routing protocol are proposed. This system in tegrates the embedded Linux system witha n ARM kernel and oc ns ists of a S3C6410 main control module a wirel ss local arean etwork WLAN card a LCD screne and so on.In the scenario of a wireless multi-hop video transmission both the H.264 and JPEG are used and their performances such as the compression rate delay and frame loss rate are analyzed in theory andc ompared in the experiment.The system is tested in the real indoor and outdoor environment.The results show that the scheme of the multi-hop video transmission experiment system can be applicable for VANET and multiple scenes and the transmission control protocol and routing protocol proposed can achieve real-time transmission and meet multi-hop requirements.

Automatic video mosaicking algorithm via dynamic key-frame

Automatic video mosaicking is a challenging task in computer vision. Current researches consider either panoramic or mapping tasks on short videos. In this paper, an automatic mosaicking algorithm is proposed for both mapping and panoramic tasks based on the adapted key-frame on videos of any length.The speeded up robust features(SURF) and the grid motion statistic(GMS) algorithm are used for feature extraction and matching between consecutive frames, which are used to compute the transformation. In order to reduce the influence of the accumulated error during image stitching, an evaluation metric is put forward for the transformation matrix. Besides, a self-growth method is employed to stitch the global image for long videos. The algorithm is evaluated by using aerial-view and panoramic videos respectively on the graphic processing unit(GPU) device, which can satisfy the real-time requirement. The experimental results demonstrate that the proposed algorithm is able to achieve a better performance than the state-of-art.

View Direction Adaptive 360 Degree Video Streaming System Based on Projected Area

360 video streaming services over the network are becoming popular. In particular, it is easy to experience 360 video through the already popular smartphone. However, due to the nature of 360 video, it is difficult to provide stable streaming service in general network environment because the size of data to send is larger than that of conventional video. Also, the real user's viewing area is very small compared to the sending amount. In this paper, we propose a system that can provide high quality 360 video streaming services to the users more efficiently in the cloud. In particular, we propose a streaming system focused on using a head mount display (HMD).

基于相机参数标定的全景监控影像拼接方法

为了提高全景监控视频的实时性,本文提出了一种基于相机标定参数的图像拼接方法。为了获取360°全景图像,该方法采用了6个固定角度的摄像头,并通过棋盘格标定板对摄像头的内外参数进行精准标定,构建出6个相机坐标系与世界坐标系的关系模型。然后通过球面投影模型,将画面投影到拼接平面上,最终达到全景监控画面的拼接效果。另外,为了更好地减少相机受外力干扰的影响,我们引入了一种参数维护机制。通过低功耗嵌入式平台和拼接服务器相结合的实验,表明该方法能够有效地提高图像拼接的速度和精度,不受环境光照条件和特征点匹配的影响,适用于无特征点或环境变化大的场景下的实时图像拼接。

全景视频实时拼接技术研究与实现

随着无人领域的不断发展,对于无人机、无人车、汽车辅助驾驶等操控者来说,需要通过无死角的全景视频进行操作,否则无法掌握无人设备完整的视野情况。通过TP2854视频数据转换芯片将AHD摄像头捕捉多路模拟视频数据转换为数字信号给处理器,再通过SURF算法和FLANN算法进行图像的特征点检测与向量计算、图像匹配、单应性矩阵计算和图像融合,最终将融合的图片数据编码和推流,实现全景视频的拼接。重点对图像融合算法不断改进,实现全景图像无拼缝、无阴影的效果。

Performance Analysis of Hybrid RR Algorithm for Anomaly Detection in Streaming Data

Automated live video stream analytics has been extensively researched in recent times.Most of the traditional methods for video anomaly detection is supervised and use a single classifier to identify an anomaly in a frame.We propose a 3-stage ensemble-based unsupervised deep reinforcement algorithm with an underlying Long Short Term Memory(LSTM)based Recurrent Neural Network(RNN).In the first stage,an ensemble of LSTM-RNNs are deployed to generate the anomaly score.The second stage uses the least square method for optimal anomaly score generation.The third stage adopts award-based reinforcement learning to update the model.The proposed Hybrid Ensemble RR Model was tested on standard pedestrian datasets UCSDPed1,USDPed2.The data set has 70 videos in UCSD Ped1 and 28 videos in UCSD Ped2 with a total of 18560 frames.Since a real-time stream has strict memory constraints and storage issues,a simple computing machine does not suffice in performing analytics with stream data.Hence the proposed research is designed to work on a GPU(Graphics Processing Unit),TPU(Tensor Processing Unit)supported framework.As shown in the experimental results section,recorded observations on framelevel EER(Equal Error Rate)and AUC(Area Under Curve)showed a 9%reduction in EER in UCSD Ped1,a 13%reduction in ERR in UCSD Ped2 and a 4%improvement in accuracy in both datasets.

Augmented virtual environment: fusion of real-time video and 3D models in the digital earth system

An Augmented virtual environment(AVE)is concerned with the fusion of real-time video with 3D models or scenes so as to augment the virtual environment.In this paper,a new approach to establish an AVE with a wide field of view is proposed,including real-time video projection,multiple video texture fusion and 3D visualization of moving objects.A new diagonally weighted algorithm is proposed to smooth the apparent gaps within the overlapping area between the two adjacent videos.A visualization method for the location and trajectory of a moving virtual object is proposed to display the moving object and its trajectory in the 3D virtual environment.The experimental results showed that the proposed set of algorithms are able to fuse multiple real-time videos with 3D models efficiently,and the experiment runs a 3D scene containing two million triangles and six real-time videos at around 55 frames per second on a laptop with 1GB of graphics card memory.In addition,a realistic AVE with a wide field of view was created based on the Digital Earth Science Platform by fusing three videos with a complex indoor virtual scene,visualizing a moving object and drawing its trajectory in the real time.

全景视频拼接技术在智能化综采工作面的应用

打破解决采煤工作面地面远程操控生产技术壁垒,保证地面远程操控生产期间安全、高效、可视化,文章基于智能化开采技术,介绍了全景视频拼接技术的技术特点、建设效果及其在综采工作面的应用和取得的效果。

Supporting multi-user real-time VBR video service using dynamic bandwidth management based on prediction

Real-time variable bit rate（VBR） video is expected to take a significant portion of multimedia applications.However,plentiful challenges to VBR video service provision have been raised for its characteristic of high traffic abruptness.To support multi-user real-time VBR video transmission with high bandwidth utilization and satisfied quality of service（QoS）,this article proposes a practical dynamic bandwidth management scheme.This scheme forecasts future media rate of VBR video by employing time-domain adaptive linear predictor and using media delivery index（MDI） as both QoS measurement and complementary management reference.In addition,to support multi-user application,an adjustment priorities classified strategy is also put forward.Finally,a test-bed based on this management scheme is established.The experimental results demonstrate that the scheme proposed in this article is efficient with bandwidth utilization increased by 20%-60% compared to a fixed service rate and QoS guaranteed.