Recent Advances in Video Coding for Machines Standard and Technologies

To improve the performance of video compression for machine vision analysis tasks,a video coding for machines(VCM)standard working group was established to promote standardization procedures.In this paper,recent advances in video coding for machine standards are presented and comprehensive introductions to the use cases,requirements,evaluation frameworks and corresponding metrics of the VCM standard are given.Then the existing methods are presented,introducing the existing proposals by category and the research progress of the latest VCM conference.Finally,we give conclusions.

Early all-zero blocks detecting method for video coding based on novel threshold

In order to decrease both computational complexity and coding time, an improved algorithm for the early detection of all-zero blocks （AZBs） in H. 264/AVC is proposed. The previous AZBs detection algorithms are reviewed. Three types of transformed frequency-domain coefficients, which are quantized to zeros, are analyzed. Based on the three types of frequencydomain scaling factors, the corresponding spatial coefficients are derived. Then the Schwarz inequality is applied to the derivation of the three thresholds based on spatial coefficients. Another threshold is set on the basis of the probability distribution of zero coefficients in a block. As a result, an adaptive AZBs detection algorithm is proposed based on the minimum of the former three thresholds and the threshold of zero blocks distribution. The simulation results show that, compared with the existing AZBs detection algorithms, the proposed algorithm achieves a 5% higher detection ratio in AZBs and 4% to 10% computation saving with only 0. 1 dB video quality degradation.

A NEW APPROACH TO DVC COMBINED WITH ROI EXTRACTION AND RESIDUAL VIDEO CODING FOR MULTIMEDIA SENSOR NETWORKS

The Wyner-Ziv distributed video coding scheme is characterized for its intraframe encoder and interframe decoder which can also approach the efficiency of an interframe encoder-decoder system. In Wyner-Ziv residual coding of video, the residual of a frame with respect to a reference frame is Wyner-Ziv encoded, which can reduces the input entropy and leads to a higher coding efficiency than directly encoding the original frame. In this paper, we propose a new approach of residual coding combined with Region Of Interest (ROI) extraction. Experimental results show that, the proposed scheme achieves better rate-distortion performance compared to conventional Wyner-Ziv coding scheme.

Perceptual video coding method based on JND and AR model

In order to achieve better perceptual coding quality while using fewer bits, a novel perceptual video coding method based on the just-noticeable-distortion （JND） model and the auto-regressive （AR） model is explored. First, a new texture segmentation method exploiting the JND profile is devised to detect and classify texture regions in video scenes. In this step, a spatial-temporal JND model is proposed and the JND energy of every micro-block unit is computed and compared with the threshold. Secondly, in order to effectively remove temporal redundancies while preserving high visual quality, an AR model is applied to synthesize the texture regions. All the parameters of the AR model are obtained by the least-squares method and each pixel in the texture region is generated as a linear combination of pixels taken from the closest forward and backward reference frames. Finally, the proposed method is compared with the H.264/AVC video coding system to demonstrate the performance. Various sequences with different types of texture regions are used in the experiment and the results show that the proposed method can reduce the bit-rate by 15% to 58% while maintaining good perceptual quality.

Overview of the Second Generation AVS Video Coding Standard(AVS2)

AVS2 is a new generation video coding standard developed by the AVS working group. Compared with the first generation AVS video coding standard, known as AVS1, AVS2 significantly improves coding performance by using many new coding technologies, e.g., adaptive block partition and two level transform coding. Moreover, for scene video, e.g. surveillance video and conference vid?eo, AVS2 provided a background picture modeling scheme to achieve more accurate prediction, which can also make object detec?tion and tracking in surveillance video coding more flexible. Experimental results show that AVS2 is competitive with High Effi?ciency Video Coding (HEVC) in terms of performance. Especially for scene video, AVS2 can achieve 39% bit rate saving over HEVC.

Color compensation for multi-view video coding based on diversity of cameras

A novel color compensation method for multi-view video coding (MVC) is proposed, which efficiently exploits the inter-view dependencies between views with the existence of color mismatch caused by the diversity of cameras. A color compensation model is developed in RGB channels and then extended to YCbCr channels for practical use. A modified inter-view reference picture is constructed based on the color compensation model, which is more similar to the coding picture than the original inter-view reference picture. Moreover, the color compensation factors can be derived in both encoder and decoder, therefore no additional data need to be transmitted to the decoder. The experimental results show that the proposed method improves the coding efficiency of MVC and maintains good subjective quality.

In-Network Puncturing for Delay-Efficient Rate Control of Distributed Video Coding in Wireless Video Sensor Networks

Wyner-Ziv Video Coding （WZVC） is considered as a promising video coding scheme for Wireless Video Sensor Networks （WVSNs） due to its high compression efficiency and error resilience functionalities, as well as its low encoding complex- ity. To achieve a good Rate-Distortion （R-D） per- formance, the current WZVC paradi^prls usually a- dopt an end-to-end rate control scheme in which the decoder repeatedly requests the additional deco- ding data from the encoder for decoding Wyner-Ziv frames. Therefore, the waiting time of the additional decoding data is especially long in multihop WVSNs. In this paper, we propose a novel pro- gressive in-network rate control scheme for WZVC. The proposed in-network puncturing-based rate control scheme transfers the partial channel codes puncturing task from the encoder to the relay nodes. Then, the decoder can request the addition- al decoding data from the relay nodes instead of the encoder, and the total waiting time for deco- ding Wyner-Ziv frames is reduced consequently. Simulation results validate the proposed rate con- trol scheme.

Cooperative Caching for Scalable Video Coding Using Value-Decomposed Dimensional Networks

Scalable video coding(SVC)has been widely used in video-on-demand(VOD)service,to efficiently satisfy users’different video quality requirements and dynamically adjust video stream to timevariant wireless channels.Under the 5G network structure,we consider a cooperative caching scheme inside each cluster with SVC to economically utilize the limited caching storage.A novel multi-agent deep reinforcement learning(MADRL)framework is proposed to jointly optimize the video access delay and users’satisfaction,where an aggregation node is introduced helping individual agents to achieve global observations and overall system rewards.Moreover,to cope with the large action space caused by the large number of videos and users,a dimension decomposition method is embedded into the neural network in each agent,which greatly reduce the computational complexity and memory cost of the reinforcement learning.Experimental results show that:1)the proposed value-decomposed dimensional network(VDDN)algorithm achieves an obvious performance gain versus the traditional MADRL;2)the proposed VDDN algorithm can handle an extremely large action space and quickly converge with a low computational complexity.

Frame-layer bit allocation for multi-view video coding based on frame complexity estimation

Current multi-view video coding （MVC） reference model in joint video team （JVT） does not provide efficient rate control schemes. This paper presents a rate control algorithm for MVC by improving the quadratic rate-distortion （R-D） model. We reasonably allocate bit-rate among views based on the correlation analysisl The proposed algorithm consists of three levels to control the rate bits more accurately, of which the frame layer allocates bits according to the frame complexity and the temporal activity. Extensive experiments show that the proposed algorithm can control the bit rate efficiently.

A WYNER-ZIV VIDEO CODING METHOD UTILIZING MIXTURE CORRELATION NOISE MODEL

In Wyner-Ziv (WZ) Distributed Video Coding (DVC), correlation noise model is often used to describe the error distribution between WZ frame and the side information. The accuracy of the model can influence the performance of the video coder directly. A mixture correlation noise model in Discrete Cosine Transform (DCT) domain for WZ video coding is established in this paper. Different correlation noise estimation method is used for direct current and alternating current coefficients. Parameter estimation method based on expectation maximization algorithm is used to estimate the Laplace distribution center of direct current frequency band and Mixture Laplace-Uniform Distribution Model (MLUDM) is established for alternating current coefficients. Experimental results suggest that the proposed mixture correlation noise model can describe the heavy tail and sudden change of the noise accurately at high rate and make significant improvement on the coding efficiency compared with the noise model presented by DIStributed COding for Video sERvices (DISCOVER).

Error concealment in pixel based Wyner-Ziv video coding

In the Wyner-Ziv(WZ) video coding paradigm, a virtual correlation channel is assumed between the quantized source and the side information(SI) at the decoder, and channel coding is applied to achieve compression. In this paper, errors caused by the virtual correlation channel are addressed and an error concealment approach is proposed for pixel-based WZ video coding. In the approach, errors after decoding are classified into two types. Type 1 errors are caused by residual bit errors after channel decoding, while type 2 errors are due to low quality of SI in part of a frame which causes SI not lying within the quantization bin of a decoded quantized pixel value. Two separate strategies are respectively designed to detect and conceal the two types of errors. Simulations are carried out and results are presented to demonstrate the effectiveness of the proposed approach.

Improving Low Bitrate Video Coding via Computation Incorporating A Priori Information

The growing number of mobile users, as well as the diversification in types of services have resulted in increasing demands for wireless network bandwidth in recent years. Although evolving transmission techniques are able to enlarge the network capacity to some degree, they still cannot satisfy the requirements of mobile users. Meanwhile, following Moore's Law, the data processing capabilities of mobile user terminals are continuously improving. In this paper, we explore possible methods of trading strong computational power at wireless terminals for transmission efficiency of communications. Taking the specific scenario of wireless video conversation, we propose a model-based video coding scheme by learning the structures in multimedia contents. Benefiting from both strong computing capability and pre-learned model priors, only low-dimensional parameters need to be transmitted; and the intact multimedia contents can also be reconstructed at the receivers in real-time. Experiment results indicate that, compared to conventional video codecs, the proposed scheme significantly reduces the data rate with the aid of computational capability at wireless terminals.

AVS 3D Video Coding Technology and System

Following the success of the audio video standard （AVS） for 2D video coding, in 2008, the China AVS workgroup started developing 3D video （3DV） coding techniques. In this paper, we discuss the background, technical features, and applications of AVS 3DV coding technology. We introduce two core techniques used in AVS 3DV coding： inter-view prediction and enhanced stereo packing coding. We elaborate on these techniques, which are used in the AVS real-time 3DV encoder. An application of the AVS 3DV coding system is presented to show the great practical value of this system. Simulation results show that the advanced techniques used in AVS 3DV coding provide remarkable coding gain compared with techniques used in a simulcast scheme.

Efficient fast mode decision using mode complexity for multi-view video coding

The variable block-size motion estimation(ME) and disparity estimation(DE) are adopted in multi-view video coding(MVC) to achieve high coding efficiency. However, much higher computational complexity is also introduced in coding system, which hinders practical application of MVC. An efficient fast mode decision method using mode complexity is proposed to reduce the computational complexity. In the proposed method, mode complexity is firstly computed by using the spatial, temporal and inter-view correlation between the current macroblock(MB) and its neighboring MBs. Based on the observation that direct mode is highly possible to be the optimal mode, mode complexity is always checked in advance whether it is below a predefined threshold for providing an efficient early termination opportunity. If this early termination condition is not met, three mode types for the MBs are classified according to the value of mode complexity, i.e., simple mode, medium mode and complex mode, to speed up the encoding process by reducing the number of the variable block modes required to be checked. Furthermore, for simple and medium mode region, the rate distortion(RD) cost of mode 16×16 in the temporal prediction direction is compared with that of the disparity prediction direction, to determine in advance whether the optimal prediction direction is in the temporal prediction direction or not, for skipping unnecessary disparity estimation. Experimental results show that the proposed method is able to significantly reduce the computational load by 78.79% and the total bit rate by 0.07% on average, while only incurring a negligible loss of PSNR(about 0.04 d B on average), compared with the full mode decision(FMD) in the reference software of MVC.

A parallel memory architecture for video coding

To efficiently exploit the performance of single instruction multiple data （SIMD） architectures for video coding, a parallel memory architecture with power-of-two memory modules is proposed. It employs two novel skewing schemes to provide conflict-free access to adjacent elements （8-bit and 16-bit data types） or with power-of-two intervals in both horizontal and vertical directions, which were not possible in previous parallel memory architectures. Area consumptions and delay estimations are given respectively with 4, 8 and 16 memory modules. Under a 0.18-pm CMOS technology, the synthesis results show that the proposed system can achieve 230 MHz clock frequency with 16 memory modules at the cost of 19k gates when read and write latencies are 3 and 2 clock cycles, respectively. We implement the proposed parallel memory architecture on a video signal processor （VSP）. The results show that VSP enhanced with the proposed architecture achieves 1.28× speedups for H.264 real-time decoding.

Very Low Bit Rate Video Coding Using Motion Compensated 3 D Wavelet Transform

A new motion compensated 3 D wavelet transform (MC 3DWT) video coding scheme is presented in this paper. The new coding scheme has a good performance in average PSNR, compression ratio and visual quality of reconstructions compared with the existing 3 D wavelet transform (3DWT) coding methods and motion compensated 2 D wavelet transform (MC WT) coding method. The new MC 3DWT coding scheme is suitable for very low bit rate video coding.

3-D Wavelet Transform for Very Low Bit-Rate Video Coding

A new improved Goh's 3 D wavelet transform(WT) coding scheme is presented in this paper. The new scheme has great advantages including a simple code structure, low computation cost and good performance in PSNR, compression ratios and visual quality of reconstructions, when compared to the other existing 3 D WT coding methods and the 2 D WT based coding methods. The new 3 D WT coding scheme is suitable for very low bit rate video coding.

Fast mode decision for Hierarchical-B pictures in scalable video coding

To decrease the computational complexity of adaptive inter-layer prediction and improve the encoding efficiency in sealable video coding, a mode decision algorithm is proposed by exploiting the part of used candidate modes of the co-located reference macrobloeks for Hierarchical-B pictures. This scheme reduces the amount of the candidate modes to generate a dynamic list for the current encoding macroblock according to the statistical information derived from the co-located reference macroblocks in different temporal levels. The experimental results show that this fast algorithm reduces approximately 31% encoding time on average with the negligible loss of encoding performance.

An Introduction to High Efficiency Video Coding Range Extensions

High Efficiency Video Coding (HEVC) is the latest international video coding standard, which can provide the similar quality with about half bandwidth compared with its predecessor, H.264/MPEG?4 AVC. To meet the requirement of higher bit depth coding and more chroma sampling formats, range extensions of HEVC were developed. This paper introduces the coding tools in HEVC range extensions and provides experimental results to compare HEVC range extensions with previous video coding standards. Ex?perimental results show that HEVC range extensions improve coding efficiency much over H.264/MPEG?4 AVC High Predictive profile, especially for 4K sequences.

Architecture of Intra Prediction for High Efficiency Video Coding

This paper explains intra prediction method for High Efficiency Video Coding(HEVC).Intra prediction removes correlation of adjacent samples in spatial domain.Intra predictor requires reference images which are stored in external memory.Memory access is required frequently in process of intra prediction.The proposed architecture can reduce external memory access by optimized internal buffer.