Real-time flow-based video abstraction using OpenCL

A non-photorealistic rendering technique is a method to show various effects different from those of realistic image generation.Of the various techniques,flow-based image abstraction displays the shape and color features well and performs a stylistic visual abstraction.But real-time rendering is impossible when CPU is used because it applies various filtering and iteration methods.In this paper,we present real-time processing methods of video abstraction using open open computing language(OpenCL),technique of general-purpose computing on graphics processing units(GPGPU).Through the acceleration of general-purpose computing(GPU),16 frame-per-second(FPS)or greater is shown to process video abstraction.

Two-Phase Rate Adaptation Strategy for Improving Real-Time Video QoE in Mobile Networks

With the popularity of smart handheld devices, mobile streaming video has multiplied the global network traffic in recent years. A huge concern of users' quality of experience(Qo E) has made rate adaptation methods very attractive. In this paper, we propose a two-phase rate adaptation strategy to improve users' real-time video Qo E. First, to measure and assess video Qo E, we provide a continuous Qo E prediction engine modeled by RNN recurrent neural network. Different from traditional Qo E models which consider the Qo E-aware factors separately or incompletely, our RNN-Qo E model accounts for three descriptive factors(video quality, rebuffering, and rate change) and reflects the impact of cognitive memory and recency. Besides, the video playing is separated into the initial startup phase and the steady playback phase, and we takes different optimization goals for each phase: the former aims at shortening the startup delay while the latter ameliorates the video quality and the rebufferings. Simulation results have shown that RNN-Qo E can follow the subjective Qo E quite well, and the proposed strategy can effectively reduce the occurrence of rebufferings caused by the mismatch between the requested video rates and the fluctuated throughput and attains standout performance on real-time Qo E compared with classical rate adaption methods.

Real-time Audio &Video Transmission System Based on Visible Light Communication

With the increasing popularity of solid sate lighting devices, Visible Light Communication (VLC) is globally recognized as an advanced and promising technology to realize short-range, high speed as well as large capacity wireless data transmission. In this paper, we propose a prototype of real-time audio and video broadcast system using inexpensive commercially available light emitting diode (LED) lamps. Experimental results show that real-time high quality audio and video with the maximum distance of 3 m can be achieved through proper layout of LED sources and improvement of concentration effects. Lighting model within room environment is designed and simulated which indicates close relationship between layout of light sources and distribution of illuminance.

Adaptive Learning Video Streaming with QoE in Multi-Home Heterogeneous Networks

In recent years,real-time video streaming has grown in popularity.The growing popularity of the Internet of Things(IoT)and other wireless heterogeneous networks mandates that network resources be carefully apportioned among versatile users in order to achieve the best Quality of Experience(QoE)and performance objectives.Most researchers focused on Forward Error Correction(FEC)techniques when attempting to strike a balance between QoE and performance.However,as network capacity increases,the performance degrades,impacting the live visual experience.Recently,Deep Learning(DL)algorithms have been successfully integrated with FEC to stream videos across multiple heterogeneous networks.But these algorithms need to be changed to make the experience better without sacrificing packet loss and delay time.To address the previous challenge,this paper proposes a novel intelligent algorithm that streams video in multi-home heterogeneous networks based on network-centric characteristics.The proposed framework contains modules such as Intelligent Content Extraction Module(ICEM),Channel Status Monitor(CSM),and Adaptive FEC(AFEC).This framework adopts the Cognitive Learning-based Scheduling(CLS)Module,which works on the deep Reinforced Gated Recurrent Networks(RGRN)principle and embeds them along with the FEC to achieve better performances.The complete framework was developed using the Objective Modular Network Testbed in C++(OMNET++),Internet networking(INET),and Python 3.10,with Keras as the front end and Tensorflow 2.10 as the back end.With extensive experimentation,the proposed model outperforms the other existing intelligentmodels in terms of improving the QoE,minimizing the End-to-End Delay(EED),and maintaining the highest accuracy(98%)and a lower Root Mean Square Error(RMSE)value of 0.001.

Wireless ad hoc video transmission:a Bayesian network-based scheme

A novel bandwidth prediction and control scheme is proposed for video transmission over an ad boc network. The scheme is based on cross-layer, feedback, and Bayesian network techniques. The impacts of video quality are formulized and deduced. The relevant factors are obtained by a cross-layer mechanism or Feedback method. According to these relevant factors, the variable set and the Bayesian network topology are determined. Then a Bayesian network prediction model is constructed. The results of the prediction can be used as the bandwidth of the mobile ad hoc network （MANET）. According to the bandwidth, the video encoder is controlled to dynamically adjust and encode the right bit rates of a real-time video stream. Integrated simulation of a video streaming communication system is implemented to validate the proposed solution. In contrast to the conventional transfer scheme, the results of the experiment indicate that the proposed scheme can make the best use of the network bandwidth; there are considerable improvements in the packet loss and the visual quality of real-time video.K

RTP payload format for H.264/SVC scalable video coding

The scalable extension of H.264/AVC, known as scalable video coding or SVC, is currently the main focus of the Joint Video Team’s work. In its present working draft, the higher level syntax of SVC follows the design principles of H.264/AVC. Self-contained network abstraction layer units (NAL units) form natural entities for packetization. The SVC specification is by no means finalized yet, but nevertheless the work towards an optimized RTP payload format has already started. RFC 3984, the RTP payload specification for H.264/AVC has been taken as a starting point, but it became quickly clear that the scalable features of SVC require adaptation in at least the areas of capability/operation point signaling and documentation of the extended NAL unit header. This paper first gives an overview of the history of scalable video coding, and then reviews the video coding layer (VCL) and NAL of the latest SVC draft specification. Finally, it discusses different aspects of the draft SVC RTP payload format, in- cluding the design criteria, use cases, signaling and payload structure.

Design and implementation of multi-hop video transmission experiment system in VANET

Abts ract A wireless mutl i-hop videot ransmission experiment system is designed and implemented for vehiculra ad-hoc networks VANET and the rt ansm ission control protocol and routing protocol are proposed. This system in tegrates the embedded Linux system witha n ARM kernel and oc ns ists of a S3C6410 main control module a wirel ss local arean etwork WLAN card a LCD screne and so on.In the scenario of a wireless multi-hop video transmission both the H.264 and JPEG are used and their performances such as the compression rate delay and frame loss rate are analyzed in theory andc ompared in the experiment.The system is tested in the real indoor and outdoor environment.The results show that the scheme of the multi-hop video transmission experiment system can be applicable for VANET and multiple scenes and the transmission control protocol and routing protocol proposed can achieve real-time transmission and meet multi-hop requirements.

View Direction Adaptive 360 Degree Video Streaming System Based on Projected Area

360 video streaming services over the network are becoming popular. In particular, it is easy to experience 360 video through the already popular smartphone. However, due to the nature of 360 video, it is difficult to provide stable streaming service in general network environment because the size of data to send is larger than that of conventional video. Also, the real user's viewing area is very small compared to the sending amount. In this paper, we propose a system that can provide high quality 360 video streaming services to the users more efficiently in the cloud. In particular, we propose a streaming system focused on using a head mount display (HMD).

Performance Analysis of Hybrid RR Algorithm for Anomaly Detection in Streaming Data

Automated live video stream analytics has been extensively researched in recent times.Most of the traditional methods for video anomaly detection is supervised and use a single classifier to identify an anomaly in a frame.We propose a 3-stage ensemble-based unsupervised deep reinforcement algorithm with an underlying Long Short Term Memory(LSTM)based Recurrent Neural Network(RNN).In the first stage,an ensemble of LSTM-RNNs are deployed to generate the anomaly score.The second stage uses the least square method for optimal anomaly score generation.The third stage adopts award-based reinforcement learning to update the model.The proposed Hybrid Ensemble RR Model was tested on standard pedestrian datasets UCSDPed1,USDPed2.The data set has 70 videos in UCSD Ped1 and 28 videos in UCSD Ped2 with a total of 18560 frames.Since a real-time stream has strict memory constraints and storage issues,a simple computing machine does not suffice in performing analytics with stream data.Hence the proposed research is designed to work on a GPU(Graphics Processing Unit),TPU(Tensor Processing Unit)supported framework.As shown in the experimental results section,recorded observations on framelevel EER(Equal Error Rate)and AUC(Area Under Curve)showed a 9%reduction in EER in UCSD Ped1,a 13%reduction in ERR in UCSD Ped2 and a 4%improvement in accuracy in both datasets.

Augmented virtual environment: fusion of real-time video and 3D models in the digital earth system

An Augmented virtual environment(AVE)is concerned with the fusion of real-time video with 3D models or scenes so as to augment the virtual environment.In this paper,a new approach to establish an AVE with a wide field of view is proposed,including real-time video projection,multiple video texture fusion and 3D visualization of moving objects.A new diagonally weighted algorithm is proposed to smooth the apparent gaps within the overlapping area between the two adjacent videos.A visualization method for the location and trajectory of a moving virtual object is proposed to display the moving object and its trajectory in the 3D virtual environment.The experimental results showed that the proposed set of algorithms are able to fuse multiple real-time videos with 3D models efficiently,and the experiment runs a 3D scene containing two million triangles and six real-time videos at around 55 frames per second on a laptop with 1GB of graphics card memory.In addition,a realistic AVE with a wide field of view was created based on the Digital Earth Science Platform by fusing three videos with a complex indoor virtual scene,visualizing a moving object and drawing its trajectory in the real time.

Supporting multi-user real-time VBR video service using dynamic bandwidth management based on prediction

Real-time variable bit rate（VBR） video is expected to take a significant portion of multimedia applications.However,plentiful challenges to VBR video service provision have been raised for its characteristic of high traffic abruptness.To support multi-user real-time VBR video transmission with high bandwidth utilization and satisfied quality of service（QoS）,this article proposes a practical dynamic bandwidth management scheme.This scheme forecasts future media rate of VBR video by employing time-domain adaptive linear predictor and using media delivery index（MDI） as both QoS measurement and complementary management reference.In addition,to support multi-user application,an adjustment priorities classified strategy is also put forward.Finally,a test-bed based on this management scheme is established.The experimental results demonstrate that the scheme proposed in this article is efficient with bandwidth utilization increased by 20%-60% compared to a fixed service rate and QoS guaranteed.

Real-time ultra-wideband video streaming in long-reach passive optical networks with wireless radiation in the 10 and 60 GHz bands

Real-time video streaming using ultra-wideband(UWB) technology is experimentally demonstrated along long-reach passive optical networks(LR-PONs) with different wired and wireless reaches. Experimental tests using external and direct modulation with UWB wireless radiation in the 10- and 60-GHz bands are performed. An ultra-bendable fiber is also considered for a last-mile distribution. The video quality at the output of the optical fiber infrastructure of the LR-PON is assessed using the error vector magnitude(EVM), and the link quality indicator(LQI) is used as a figure of merit after wireless radiation. An EVM below –17 dB is achieved for both externally and directly modulated LR-PONs comprising up to 125 km of optical fiber. EVM improvement is observed for longer LR-PONs when directly modulated lasers(DMLs) are used because of the amplitude gain provided by the combined effect of dispersion and DML's chirp. Compared with optical back-to-back operation, the LQI level degrades to the maximum around 20% for LR-PONs ranging between 75 and 125 km of fiber reach and with a wireless coverage of 2 m in the 10-GHz UWB band. The same level of LQI degradation is observed using the 60-GHz UWB band with a LR-PON integrating 101 km of access network, a last-mile distribution using ultra-bendable fiber, and a 5.2-m wireless link.

利用情感激励提取足球视频精彩镜头

足球视频精彩镜头的提取是足球视频检索和摘要生成的关键技术,有着很高的学术研究价值和广泛的应用前景.在系统分析现有足球视频精彩镜头提取方法的基础上,从观众情绪波动的视角出发,以Hanjalic情感曲线的理论思想为基础,设计了一个基于情感激励模型提取足球视频精彩镜头的系统.针对足球视频构建了一种新的特征——镜头激烈度,用此特征取代运动强度特征建立情感激励模型,从而改进了Hanjalic情绪激励曲线的生成方法,提高了系统的查全率、准确率和计算性能.结合足球领域知识,实现了更准确的精彩镜头定位方法,可以根据用户的实际观看时间自动筛选精彩镜头.实验结果验证了算法具有很好的检测效果.

视频摘要技术综述

基于内容的视频分析、检索与索引技术是当前多媒体领域研究的热点之一。视频摘要主要分为两类:静态视频摘要模式(视频略览),包括基于MPEG-1/2的关键帧表示和MPEG-4的关键对象表示;动态视频摘要模式(视频梗概),包括视频概述和精彩片段。该文介绍了视频摘要的新方向——基于情感、心理学等视频摘要模型。

层次化新闻视频处理框架的设计与实现

提出了一个通用的层次化新闻视频处理框架,将新闻视频处理分为句法分段、语义标注以及视频摘要三个层次,并给出了三个层次中涉及的故事单元探测、字幕探测、视频摘要等关键技术的解决方案。框架突破了传统的新闻视频处理框架仅局限于句法分段以及单媒体特征进行处理的缺陷,通过对视音频特征进行多模态的综合分析来获取新闻视频高层的语义内容。实验通过一个新闻视频处理原型系统NVPS验证了框架的可行性,重点对故事单元探测、标题探测以及口播帧探测三个算法进行了实验,实验结果分别达到88%,86%和86%的探测准确率,从而进一步证实了层次框架在新闻视频处理方面的有效性。

基于目标检测及跟踪的视频摘要技术研究

为了能够在不丢失有用信息的前提下对海量监控视频进行摘要,提出一种基于目标检测及跟踪的视频摘要技术。首先根据混合高斯模型对视频进行背景建模,获得监控视频的背景;然后通过背景减法对运动目标进行检测,得到前景运动目标;其次对检测到的运动目标采用分层关联的思想进行目标跟踪,得到运动目标的完整信息;最后将这些运动目标及视频背景重新组装成摘要视频。实验结果表明,该方法能够有效地对监控视频进行浓缩,形成的摘要视频能够完整地保存原有视频信息,减少了存储空间,降低了成本,方便了相关人员及时获取有用信息,提高了工作效率。

视频摘要技术研究

对视频摘要技术进行了研究,将视频摘要按其表现形式分为以标题、关键帧和故事板为代表的静态视频摘要以及以缩略视频为代表的动态视频摘要,并对其中涉及的关键技术进行了探讨,最后对视频摘要技术未来的发展方向进行了总结与展望。

一种新的动态视频摘要生成方法

本文提出了一种概要序列形式的视频摘要生成方法.在视频结构化的基础上,从减少信息冗余和保持本质内容两个角度出发,分别定义了镜头(场景)的冗余度和语义重要度,据此推导出视频内容缩减和保持的原则;之后,综合可理解性的要求,给出了一种自上而下的视频摘要生成方法.以电影动作片为例的实验结果证实了方法的有效性.

面向事件的影片摘要生成方法

对视频摘要的研究已成为视频应用领域十分活跃的课题。为了获得更加实用的视频摘要,在介绍视频摘要概念和用途的基础上,依据目前的研究状况和影片类视频的特点,提出了一种适合于故事类影片的面向事件的影片摘要生成方法,并首先对该方法的细节进行了描述。该方法还使用了一种场景重要程度的评价算法,此算法的特点在于综合利用了影片中的多种媒体特征,其不仅考虑了运动特性,还考虑了感人内容对场景重要程度的影响。通过对摘要进行可用度测试和分析的结果,表明,此种方法生成的影片摘要具有较好的可用度。

基于关键帧的视频摘要研究

提出基于DirectShow组件结构实时提取各类音视频特征的方法,并综合利用这些特征,通过对常用的视频关键帧提取算法进行分析比较,提出了基于FCNN模型的自适应关键帧提取方法,该方法可以适应于各种视频类型。以关键帧为基础,提出利用插值算法建立视频摘要的方法。最后以新闻视频为例,给出了基于内容的新闻视频摘要系统(KBVA-1)。