Interactive transport of multi-view videos for 3DTV applications

The authors propose a novel method for transporting multi-view videos that aims to keep the bandwidth requirements on both end-users and servers as low as possible. The method is based on application layer multicast, where each end point re- ceives only a selected number of views required for rendering video from its current viewpoint at any given time. The set of selected videos changes in real time as the user’s viewpoint changes because of head or eye movements. Techniques for reducing the black-outs during fast viewpoint changes were investigated. The performance of the approach was studied through network experiments.

Multi-view video color correction using dynamic programming

Color inconsistency between views is an important problem to be solved in multi-view video systems. A multi-view video color correction method using dynamic programming is proposed. Three-dimensional histograms are constructed with sequential conditional probability in HSI color space. Then, dynamic programming is used to seek the best color mapping relation with the minimum cost path between target image histogram and source image histogram. Finally, video tracking technique is performed to correct multi-view video. Experimental results show that the proposed method can obtain better subjective and objective performance in color correction.

Color compensation for multi-view video coding based on diversity of cameras

A novel color compensation method for multi-view video coding (MVC) is proposed, which efficiently exploits the inter-view dependencies between views with the existence of color mismatch caused by the diversity of cameras. A color compensation model is developed in RGB channels and then extended to YCbCr channels for practical use. A modified inter-view reference picture is constructed based on the color compensation model, which is more similar to the coding picture than the original inter-view reference picture. Moreover, the color compensation factors can be derived in both encoder and decoder, therefore no additional data need to be transmitted to the decoder. The experimental results show that the proposed method improves the coding efficiency of MVC and maintains good subjective quality.

Frame-layer bit allocation for multi-view video coding based on frame complexity estimation

Current multi-view video coding （MVC） reference model in joint video team （JVT） does not provide efficient rate control schemes. This paper presents a rate control algorithm for MVC by improving the quadratic rate-distortion （R-D） model. We reasonably allocate bit-rate among views based on the correlation analysisl The proposed algorithm consists of three levels to control the rate bits more accurately, of which the frame layer allocates bits according to the frame complexity and the temporal activity. Extensive experiments show that the proposed algorithm can control the bit rate efficiently.

Temporally Consistent Depth Map Estimation for 3D Video Generation and Coding

In this paper, we propose a new algorithm for temporally consistent depth map estimation to generate three-dimensional video. The proposed algorithm adaptively computes the matching cost using a temporal weighting function, which is obtained by block-based moving object detection and motion estimation with variable block sizes. Experimental results show that the proposed algorithm improves the temporal consistency of the depth video and reduces by about 38% both the flickering artefact in the synthesized view and the number of coding bits for depth video coding.

Efficient fast mode decision using mode complexity for multi-view video coding

The variable block-size motion estimation(ME) and disparity estimation(DE) are adopted in multi-view video coding(MVC) to achieve high coding efficiency. However, much higher computational complexity is also introduced in coding system, which hinders practical application of MVC. An efficient fast mode decision method using mode complexity is proposed to reduce the computational complexity. In the proposed method, mode complexity is firstly computed by using the spatial, temporal and inter-view correlation between the current macroblock(MB) and its neighboring MBs. Based on the observation that direct mode is highly possible to be the optimal mode, mode complexity is always checked in advance whether it is below a predefined threshold for providing an efficient early termination opportunity. If this early termination condition is not met, three mode types for the MBs are classified according to the value of mode complexity, i.e., simple mode, medium mode and complex mode, to speed up the encoding process by reducing the number of the variable block modes required to be checked. Furthermore, for simple and medium mode region, the rate distortion(RD) cost of mode 16×16 in the temporal prediction direction is compared with that of the disparity prediction direction, to determine in advance whether the optimal prediction direction is in the temporal prediction direction or not, for skipping unnecessary disparity estimation. Experimental results show that the proposed method is able to significantly reduce the computational load by 78.79% and the total bit rate by 0.07% on average, while only incurring a negligible loss of PSNR(about 0.04 d B on average), compared with the full mode decision(FMD) in the reference software of MVC.

CODING-ORIENTED MULTI-VIEW VIDEO COLOR CORRECTION

Color inconsistency between views is an important problem to be solved in multi-view video applications, such as free viewpoint television and other three-dimensional video systems. In this paper, by combining with multi-view video coding, a coding-oriented multi-view video color correction method is proposed. We first separate foreground and background in first Group Of Pictures （GOP） by using SKIP coding mode. Then by transferring means and standard deviations in backgrounds, color correction is performed for each frame in GOP, and multi-view video coding is performed and used to renew the backgrounds. Experimental results ances in color correction and multi-view video show the proposed method can obtain better performcoding.

Trifocal tensor based side information generation for multi-view distributed video code

Distributed video coding （DVC） is a new video coding approach based on Wyner-Ziv theorem. The novel uplink-friendly DVC, which offers low-complexity, low-power consuming, and low-cost video encoding, has aroused more and more research interests. In this paper a new method based on multiple view geometry is presented for spatial side information generation of uncalibrated video sensor network. Trifocal tensor encapsulates all the geometric relations among three views that are independent of scene structure; it can be computed from image correspondences alone without requiring knowledge of the motion or calibration. Simulation results show that trifocal tensor-based spatial side information improves the rate-distortion performance over motion compensation based interpolation side information by a maximum gap of around 2dB. Then fusion merges the different side information （temporal and spatial） in order to improve the quality of the final one. Simulation results show that the rate-distortion gains about 0.4 dB.

A Tensor-based Enhancement Algorithm for Depth Video

In order to repair the dark holes in Kinect depth video, we propose a depth hole-filling method based on tensor.First, we process the original depth video by a weighted moving average system. Then, reconstruct the low-rank sensors and sparse sensors of the video utilize the tensor recovery method, through which the rough motion saliency can be initially separated from the background. Finally, construct a four-order tensor for moving target part, by grouping similar patches. Then we can formulate the video denoising and hole filling problem as a low-rank completion problem. In the proposed algorithm, the tensor model is used to preserve the spatial structure of the video modality. And we employ the block processing method to overcome the problem of information loss in traditional video processing based on frames. Experimental results show that our method can significantly improve the quality of depth video, and has strong robustness.

基于时空流特征融合的俯视视角下奶牛跛行自动检测方法

[目的/意义]奶牛跛行检测是规模化奶牛养殖过程中亟待解决的重要问题,现有方法的检测视角主要以侧视为主。然而,侧视视角存在着难以消除的遮挡问题。本研究主要解决侧视视角下存在的遮挡问题。[方法]提出一种基于时空流特征融合的俯视视角下奶牛跛行检测方法。首先,通过分析深度视频流中跛行奶牛在运动过程中的位姿变化,构建空间流特征图像序列。通过分析跛行奶牛行走时躯体前进和左右摇摆的瞬时速度,利用光流捕获奶牛运动的瞬时速度,构建时间流特征图像序列。将空间流与时间流特征图像组合构建时空流融合特征图像序列。其次,利用卷积块注意力模块(Convolutional Block Attention Module, CBAM)改进PP-TSMv2 (PaddlePaddle-Temporal Shift Module v2)视频动作分类网络,构建奶牛跛行检测模型Cow-TSM (Cow-Temporal Shift Module)。最后,分别在不同输入模态、不同注意力机制、不同视频动作分类网络和现有方法 4个方面对比,进行奶牛跛行实验,以探究所提出方法的优劣性。[结果和讨论]共采集处理了180段奶牛图像序列数据,跛行奶牛与非跛行奶牛视频段数比例为1∶1,所提出模型识别精度达到88.7%,模型大小为22 M,离线推理时间为0.046 s。与主流视频动作分类模型TSM、PP-TSM、PP-TSMv2、SlowFast和TimesFormer模型相比,综合表现最好。同时,以时空流融合特征图像作为输入时,识别精度分别比单时间模态与单空间模态分别提升12%与4.1%,证明本研究中模态融合的有效性。通过与通道注意力(Squeeze-and-Excitation, SE)、卷积核注意力(Selective Kernel, SK)、坐标注意力(Coordinate Attention, CA)与CBAM不同注意力机制进行消融实验,证明利用CBAM注意力机制构建奶牛跛行检测模型效果最佳。最后,与现有跛行检测方法进行对比,所提出的方法同时具有较好的性能和实用性。[结论]本研究能够避免侧视视角下检测跛行奶牛时出现的遮挡问题,对于减少奶牛跛行发生率、提高牧场经济效益具有重要意义,符合牧场规模化建设的需求。

智能技术支持的教师专业发展和课堂教学——“AI新热潮之下的冷思考与新出发”研讨会青年学者论坛综述

人工智能作为新兴科技的标志性内容,在成为经济发展的新引擎、国际竞争的新焦点、社会建设新机遇的同时,也成为了促进教育变革的重要力量。目前,以元宇宙、ChatGPT等为代表的技术正在全球掀起新一轮的智能科技浪潮,也将对教育场景产生新一轮的冲击。基于此,以“人工智能促进未来教育发展:AI新热潮之下的冷思考与新出发”为主题的研讨会意图回应和探究热潮之下的技术走向,并探讨新的技术发展对未来教育的持续及颠覆性影响。围绕“智能技术支持下的教师专业发展和课堂教学”这一研究主题,来自四所不同高校的青年研究者在此次研讨会中分别基于不同的研究场景,从技术设计和实证研究等层面呈现了人工智能赋能教师专业发展的潜在探索维度和未来发展方向。希望通过介绍四个学术报告,带领大家认识人工智能赋能教师专业发展的潜在研究与实践样态,也希望引发大家对智能技术支持下的教师专业发展和课堂教学这一研究主题进行持续且更加深入的思考,为我国人工智能赋能教师专业发展的事业贡献力量。

超高清视频技术要素主观评价研究

本文研究了超高清视频技术要素对主观体验的影响,通过主观评价实验量化了分辨率、动态范围、帧率和量化深度等技术要素的提升对观看体验的贡献。实验结果显示,技术要素的提升对改善主观体验有积极作用,其中,4K分辨率、HDR技术、高帧率对主观体验的影响较为显著,2K HDR在某些情况下优于4K SDR。该研究为超高清视频技术的发展提供了参考。

VideoLog可视化测井油管接箍自动识别方法

在可视化测井中,深度对于判断油管缺陷位置至关重要,而现有的测深系统具有一定的深度误差。实际工程中,可通过识别油管接箍再参照油管数据表来准确标定仪器的深度。本文基于运动视频图像处理,提出了一种油管接箍自动识别方法。利用VideoLog可视化测井系统采集井下油管视频图像,通过对视频图像进行形态学处理、特征参数提取、接箍判决等过程来准确识别接箍。实验结果表明,同一个接箍在视频中会多次出现,也会被多次识别到,同一接箍平均识别率为86.9%,接箍计数的正确率为100%。方法已成功用于可视化测井视频解释处理中,取得了较好的工程应用效果。

基于depth-map和分布式视频编码的多视点视频传输方法

针对多视点视频传输系统数据量庞大的问题,提出一种基于深度图(depth-map)和分布式视频编码(DVC)的不等错误保护(UEP)传输方法。该方法首先基于多个视点提取深度图;然后,在传输过程中传输一个视点及其深度图;最后,经过网络传输,在解码端由一个视点图及其深度图生成其他视点。由于视点图和深度图在解码端的重要程度不同,对需要传输的视点图和深度图采用不同的分布式视频编码方法,进行不平等的错误保护。仿真实验结果表明,所提传输方法比传统的分布式多视点视频编码传输系统具有更好的抗误码性能,提高了传输可靠性,图像的峰值信噪比(PSNR)约提高1.5 dB。

INPAINTING ALGORITHM FOR KINECT DEPTH MAP BASED ON FOREGROUND SEGMENTATION

The depth information of the scene indicates the distance between the object and the camera,and depth extraction is a key technology in 3D video system.The emergence of Kinect makes the high resolution depth map capturing possible.However,the depth map captured by Kinect can not be directly used due to the existing holes and noises,which needs to be repaired.We propose a texture combined inpainting algorithm in this paper.Firstly,the foreground is segmented combined with the color characteristics of the texture image to repair the foreground of the depth map.Secondly,region growing is used to determine the match region of the hole in the depth map,and to accurately position the match region according to the texture information.Then the match region is weighted to fill the hole.Finally,a Gaussian filter is used to remove the noise in the depth map.Experimental results show that the proposed method can effectively repair the holes existing in the original depth map and get an accurate and smooth depth map,which can be used to render a virtual image with good quality.

Bandwidth-Efficient Transmission Method for User View-Oriented Video Services

The trend in video viewing has been evolving beyond simply providing a multi-view option.Recently,a function that allows selection and viewing of a clip from a multi-view service that captures a specific range or object has been added.In particular,the free-view service is an extended concept of multi-view and provides a freer viewpoint.However,since numerous videos and additional data are required for its construction,all of the clips constituting the content cannot be simultaneously provided.Only certain clips are selected and provided to the user.If the video is not the preferred video,change request is made,and a delay occurs during retransmission from the server.Delays due to frequent re-requests degrade the overall quality of service.For free-view services,selectively transmitting the video according to the user’s desired viewpoint and region of interest within the limited network of available videos is important.In this study,we propose a method of screening and providing the correct video based on objects in the contents.Based on the method of recognizing the object in each clip,we designed a method of setting its priority based on information about the object’s location for each viewpoint.During the transmission and receiving process using this information,the selected video can be rapidly recognized and changed.Herein,we present a service system configuration method and propose video selection examples for free-view services.

3DV quality model based depth maps for view synthesis in FTV system

Depth maps are used for synthesis virtual view in free-viewpoint television （FTV） systems. When depth maps are derived using existing depth estimation methods, the depth distortions will cause undesirable artifacts in the synthesized views. To solve this problem, a 3D video quality model base depth maps （D-3DV） for virtual view synthesis and depth map coding in the FTV applications is proposed. First, the relationships between distortions in coded depth map and rendered view are derived. Then, a precisely 3DV quality model based depth characteristics is develop for the synthesized virtual views. Finally, based on D-3DV model, a multilateral filtering is applied as a pre-processed filter to reduce rendering artifacts. The experimental results evaluated by objective and subjective methods indicate that the proposed D-3DV model can reduce bit-rate of depth coding and achieve better rendering quality.

Wedge template optimization and parallelization of depth map in intra-frame prediction algorithms

To reduce the computational complexity and storage cost caused by wedge segmentation algorithm,a scheme of simplifying wedge matching is proposed.It takes advantage of the correlation of the wedge separation line of depth map and the direction of intra-prediction for 3D high-efficiency video coding(3D-HEVC).According to the difference of wedge segmentation between adjacent edge and opposite edge,a set only including 104×4 wedgelet templates is given.By expanding of the wedge wave of a certain minimum unit,a simple separation line acquisition method for different size of depth block is put forward.Furthermore,based on the array processor(DPR-CODEC)developed by project team,an efficient parallel scheme of the improved wedge segmentation mode prediction is introduced.By the scheme,prediction unit(PU)size can be changed randomly from 4×4 to 8×8,16×16,and 32×32,which is more in line with the needs of the HEVC standard.Veri-fied with test sequence in HTM16.1 and the Xilinx virtex-6 field programmable gate array(FPGA)respectively,the experiment results show that the proposed methods save 99.2%of the storage space and 63.94%of the encoding time,the serial/parallel acceleration ratio of each template reaches 1.84 in average.The coding performance,storage and resource consumption are considered for both.

基于云边协同的煤矿井下尺度自适应目标跟踪方法

煤矿井下监控视频中的运动目标通常存在较大的尺度变化和形变,导致基于计算机视觉的目标跟踪算法准确率不高,且海量的视频数据导致基于云端的集中式数据处理方式难以满足目标跟踪的实时性要求。针对上述问题,提出了一种基于云边协同的煤矿井下尺度自适应目标跟踪方法。设计了基于深度估计的尺度自适应目标跟踪算法,通过构建深度-尺度估计模型,利用目标深度值估计尺度值,实现尺度自适应目标跟踪,解决了目标尺度变化和形变导致跟踪准确率不高的问题;设计了一种基于云边协同的智能监控系统架构,将尺度自适应目标跟踪算法细粒度划分后的子模块按所需计算资源分别部署在系统的边缘端和云端,通过边缘端和云端的分布式并行处理提高算法运行效率,解决了集中式数据处理方式实时性差的问题。将基于云边协同的煤矿井下尺度自适应目标跟踪方法应用于煤矿井下视频序列,对其跟踪性能和实时性能进行实验验证,结果表明:与核相关滤波(KCF)、判别型尺度空间跟踪(DSST)算法、基于多特征融合的尺度自适应(SAMF)算法3种经典目标跟踪算法相比,基于深度估计的尺度自适应目标跟踪算法在煤矿井下目标出现较大尺度变化和形变时,具有更高的跟踪精度和成功率;与传统的云计算处理方式相比,基于云边协同的尺度自适应目标跟踪算法部署方式使算法总时延降低了32.55%,有效提升了煤矿井下智能监控系统目标跟踪的实时性能。

Depth Enhancement Methods for Centralized Texture-Depth Packing Formats

To deliver three-dimension （3D） videos through the current two-dimension （2D） broadcasting systems, the frame-compati-ble packing formats properly including one texture frame and one depth map in various down-sampling ratios have been proposed to achieve the simplest and most effective solution. To enhance the compatible centralized texture-depth packing （CTDP） formats, in this paper, we further introduce two depth enhancement algorithms to further improve the quality of CT-DP formats for delivering 3D video services. To compensate the loss of color YCbCr 444 to 420 conversion of colored-depth, two efficient depth reconstruction processes based on texture and depth consistency are proposed. Experimental re-sults show that the proposed enhanced CTDP depacking pro-cess outperforms the 2DDP format and the original CTDP de-packing procedure in synthesizing virtual views. With the help of the proposed efficient depth reconstruction processes, more correct reconstructed depth maps and better synthesized quality can be achieved. Before the available 3D broadcasting systems, which adopt truly depth and texture dependent cod-ing procedure, we believe that the proposed CTDP formats with depth enhancement could help to deliver 3D videos in the current 2D broadcasting systems simply and efficiently.