MULTI-EPIPOLAR LINES MATCHING-BASED RAY-SPACE INTERPOLATION FOR FREE VIEWPOINT VIDEO SYSTEM

Ray-space based arbitrary viewpoint rendering without complex object segmentation or model construction is the main technology to realize Free Viewpoint Video(FVV) system for complex scenes. Ray-space interpolation and compression are two key techniques for the solution. In this paper,correlation among multiple epipolar lines in ray-space data is analyzed,and a new method of ray-space interpolation with multi-epipolar lines matching is proposed. Comparing with the pixel-based matching interpolation method and the block-based matching interpolation method,the proposed method can achieve higher Peak Signal to Noise Ratio(PSNR) in interpolating rayspace data and rendering arbitrary viewpoint images.

On-line Free-viewpoint Video:From Single to Multiple View Rendering

In recent years, many image-based rendering techniques have advanced from static to dynamic scenes and thus become video-based rendering （VBR） methods. But actually, only a few of them can render new views on-line. We present a new VBR system that creates new views of a live dynamic scene. This system provides high quality images and does not require any background subtraction. Our method follows a plane-sweep approach and reaches real-time rendering using consumer graphic hardware, graphics processing unit （GPU）. Only one computer is used for both acquisition and rendering. The video stream acquisition is performed by at least 3 webcams. We propose an additional video stream management that extends the number of webcams to 10 or more. These considerations make our system low-cost and hence accessible for everyone. We also present an adaptation of our plane-sweep method to create simultaneously multiple views of the scene in real-time. Our system is especially designed for stereovision using autostereoscopic displays. The new views are computed from 4 webcams connected to a computer and are compressed in order to be transfered to a mobile phone. Using GPU programming, our method provides up to 16 images of the scene in real-time. The use of both GPU and CPU makes this method work on only one consumer grade computer.

论FVV交互体验的网络需求

自由视角视频(Free Viewpoint Video,FVV)通过为观众提供360°视角转动观看/动态360°“时间凝固”等颠覆性特性,实现了观众的“导播梦”,极大地丰富了视频体验效果。本文介绍了互动视频发展历程、FVV标准化进展和典型FVV系统(系统架构、网络组成及用户体验需求),重点围绕用户自主切换视角的响应时延与网络需求展开了定量研究,最后对主要研究结果进行了总结。

FTV系统中面向虚拟视点合成的深度编码

为了降低自由视点电视系统中由于深度估计不准确和深度数据压缩带来的量化失真对重建虚拟视点图像质量产生的影响,提出了一种面向虚拟视点绘制的深度压缩算法.根据深度数据的统计特性,设计了一种基于三次B样条函数的上下采样方法对深度图进行下采样.采用多视点视频编码对下采样深度图进行编码,在解码端则对深度图进行上采样,并针对解码深度图边缘处的模糊现象,用双边滤波器优化深度图边界.实验结果表明,与传统的MVC编码方案相比,该方案在保持虚拟绘制图像质量不变(甚至略有提升)的情况下,显著降低了传输码率,有助于在自由视点系统的接收端获得高精度的重建图像.

基于自由立体视频的露天开采虚拟现实的研究与实现

文章结合最新的交互式自由立体视频技术与露天开采虚拟现实技术,设计并实现了一种基于自由立体视频的露天矿开采虚拟现实技术的交互与演示系统。文章利用3DS Max和VRMap软件建立了某露天矿区的虚拟视觉模型,并将这些虚拟视觉模型按照适合立体显示的条件进行场景渲染和视频图象生成,最后结合虚拟场景的三维显示特性设计了交互式自由立体视频软件,通过该软件展示了虚拟现实场景。该系统克服了传统虚拟现实演示的局限性,用较低的成本支持更多的用户进行参与和交互,同时也具有较强的环境拓展性。

基于特征点方向的光线-空间插值方法

基于光线-空间片内方向性特征分析,提出了适合于光线-空间的插值方法,仅对光线-空间片中位于具有明显强度变化特征的区域中的特征点计算其所在光线的方向,其余各点的方向由特征点方向插值得到,以减少方向搜索与检测的计算量,提高方向检测的准确性.在此基础上,对稀疏光线-空间片沿光线方向进行线性插值.实验结果表明,该方法计算量小,插值生成的光线-空间数据信噪比高,绘制的任意视点图像的主客观质量高.

3D自由视点视频技术及其在中国馆中的应用

阐述了3D自由视点视频应用系统相关的关键技术,包括显示、内容获取与生成、编解码重构等,并对大屏幕3D自由视点视频技术在2010年上海世博会中国国家馆中的成功应用作了简要介绍。

利用深度信息增强的自由视点DIBR技术研究

针对现有DIBR虚拟视点图像渲染算法存在的问题,提出一种基于深度信息进行后处理的DIBR虚拟视点渲染算法.该算法利用摄像机参数和深度信息,将参考视点图像的纹理和深度图变形到虚拟视点,利用基于深度信息的后处理消除变形图像中的裂缝和伪影,结合深度信息采用inpaint技术填补剩余的纹理空洞.实验结果显示,该方法不论从主观质量或是客观质量上均优于现有的虚拟视点生成方法,且计算量也不高.在数值试验中,还对多视点渲染算法的图像合成质量进行比较,对视频压缩的影响进行了分析.

一种自由视点视频系统中立体图像质量的评价方法

提出了一种自由视点视频系统中立体图像的质量评价方法。分别对参考图像与失真图像进行小波分解后用对比敏感度函数加权,计算两者之间的平均结构相似度得到基于内容的质量分数;在双目匹配区域中找到中心凹区域,在该区域上计算图像之间的平均相位相似度,得到基于深度感知的质量分数;利用阈值提取匹配区域中为人眼所感知的失真区域,与利用视差映射过来的另一视点图像的相对应区域计算平均结构相似度得到基于双目特性的质量分数。融合各部分质量分数得到最终图像质量。实验结果表明,本算法评价结果与主观感知一致性较好,更符合人眼视觉特性。

基于图割的深度图去噪算法

多视点视频编码给深度图带来的编码块效应,导致合成的虚拟视点中出现严重的几何失真,为去除深度图中编码带来的噪声,提出一种基于图割的深度图去噪算法。将深度图及相应的彩色图的局部二值模式(LBP)算子作为去噪的约束,融入算法中,有效保持深度图的边缘信息。实验结果表明,由去噪后深度图绘制的虚拟视点的质量得到大幅提高,相对目前优秀的滤波方法能平均获得0.35dB的PSNR增益,主观效果的提升非常明显。

一种快速参数估计的多视点图像颜色校正方法

提出了一种基于快速确定参数的全局直方图匹配的颜色校正方法,对于标准的多视点视频序列,利用直方图,分别快速估算出其在RGB,YUV,CIELab这3个颜色空间下的3个分量线性方程的参数值,在MATLAB环境下仿真出3个不同颜色空间中的校正结果,并对其进行分析和比较,实验结果表明了该方法的有效性和可靠性。

三维视频深度图像处理及其ASIC实现

基于自适应色度分割方法,采用专用集成电路(ASIC)完成深度图像的处理及优化。系统级仿真验证结果表明,该深度图像处理方法具有实时性、兼容性、实用性等特点,适用于实时自由视点3D视频的处理。

Hyper 360—Towards a Unified Tool Set Supporting Next Generation VR Film and TV Productions

We describe four fundamental challenges that complex real-life Virtual Reality (VR) productions are facing today (such as multi-camera management, quality control, automatic annotation with cinematography and 360˚?depth estimation) and describe an integrated solution, called Hyper 360, to address them. We demonstrate our solution and its evaluation in the context of practical productions and present related results.

基于5G的自由视角交互直播视频方案的设计与实现

5G网络为视频传播提供了良好的载体,促进了超高清视频的快速发展,并融合出多种新的应用场景,用户体验得到极大优化和提升。本文分析了5G网络、自由视角、边缘计算等关键技术,论述了采用5G+自由视角交互直播视频的整体解决方案,并介绍了应用案例及其推广情况。

Non-reference virtual view quality evaluation of MVD

The quality of virtual view based on multi-view video （MVD） plus depth format is often evaluated by PSNR or subjectively judged. However, due to synthesizing arbitrary view images, the virtual view images mostly have no reference images and are only assessed using non-reference. Virtual view images synthesized by depth estimation reference software （DERS） and view synthesis reference software （VSRS） often accompanied with blockiness and other distortions on the edge. In addition, matching level for the depth map and the corresponding texture maps of left and right views also affects the quality of the virtual view. This paper compares the edge similarity of the depth and the corresponding texture maps which generate the intermediate virtual view and combined with the virtual view＇s blockiness which causing blur to evaluate the quality of the virtual view. Experiment results show that the proposed method can reflect the quality of the virtual view better.

3DV quality model based depth maps for view synthesis in FTV system

Depth maps are used for synthesis virtual view in free-viewpoint television （FTV） systems. When depth maps are derived using existing depth estimation methods, the depth distortions will cause undesirable artifacts in the synthesized views. To solve this problem, a 3D video quality model base depth maps （D-3DV） for virtual view synthesis and depth map coding in the FTV applications is proposed. First, the relationships between distortions in coded depth map and rendered view are derived. Then, a precisely 3DV quality model based depth characteristics is develop for the synthesized virtual views. Finally, based on D-3DV model, a multilateral filtering is applied as a pre-processed filter to reduce rendering artifacts. The experimental results evaluated by objective and subjective methods indicate that the proposed D-3DV model can reduce bit-rate of depth coding and achieve better rendering quality.

Viewpoint Manipulation for Interactive Television by Using Human Pose Estimation

In current interactive television schemes, the viewpoints should be manipulated by the user. However, there is no efficient method, to assist a user in automatically identifying and tracking the optimum viewpoint when the user observes the object of interest because many objects, most often humans, move rapidly and frequently. This paper proposes a novel framework for determining and tracking the virtual camera to best capture the front of the person of interest (PoI). First, one PoI is interactively chosen in a segmented 3D scene reconstructed by space carving method. Second, key points of the human torso of the PoI are detected by using a model-based method and the human's global motion including rotation and translation is estimated by using a close-formed method with 3 corresponding points. At the last step, the front direction of PoI is tracked temporally by using the unscented particle filter (UPF). Experimental results show that the method can properly compute the front direction of the PoI and robustly track the best viewpoints.

基于自适应窗口的深度视频预处理算法

在自由视点视频系统中,由深度估计算法得到的深度视频并不精确,对虚拟视点的绘制质量和深度视频的编码效率有很大的影响。本文提出了一种基于自适应窗口的深度视频预处理算法来提高深度视频的压缩效率和虚拟视点的绘制质量。首先,将深度视频划分为不连续区域、连续区域中的前景区域和连续区域中的背景区域。然后,提取出深度视频各区域中的对象边界。最后对不连续区域中的非边界区域进行高斯滤波,降低深度值骤变对虚拟视点绘制质量的影响,对连续区域中的非边界区域采用自适应窗口滤波平滑处理以提高压缩效率。实验结果表明,提出的算法可以使深度视频的编码码率节省8.33～34.39%,同时可使绘制的虚拟视点质量平均提高0.21dB。和Silva的算法相比,在更具实际意义的低码率端编码效率更显著,绘制结果中对象边界得到了更好的保护。

面向虚拟视点图像绘制的深度图编码算法

针对不同区域对绘制虚拟视点图像质量产生不同的影响,以及深度估计不准确导致时域抖动效应影响压缩效率的问题,提出了一种面向虚拟视点图像绘制的深度图压缩算法。通过彩色图像帧差、深度图边缘提取等相关处理过程,提取深度图的静态区域、边缘区域以及动态区域。对深度图边缘区域使用了较低的量化系数,以提高深度图边缘区域编码质量;根据深度图各个区域的编码模式特点,仅对部分编码模式而不是所有模式进行率失真优化搜索,以提高深度图的编码速度;对于深度图P帧的静态区域,合理地采用了SKIP模式,以消除由于深度估计算法的局限性导致时域抖动效应对深度图压缩的影响。实验结果表明,与传统的H.264编码方案相比,本文方案在传输码流大小基本不变的前提下提高了最终虚拟视点图像边缘区域的绘制质量,其余区域主观质量相近,而深度图编码时间则节省了约77～87%。

快速3维坐标变换的绘制算法

目的基于深度图的绘制(DIBR)是一种新型的虚拟视点生成技术,在诸多方面得到了广泛的应用。然而,该技术还不能满足实时性的绘制需求。为了在保证绘制质量不下降的前提下,尽可能地提高绘制速度,提出了一种高效的3D-Warping(3维坐标变换)算法。方法主要在以下3个方面进行了改进:1)引入了深度—视差映射表技术,避免了重复地进行视差求取操作。2)对深度平坦的像素块进行基于块的3D-Warping,减少了映射的次数。对深度非平坦像素块中的像素点采取传统的基于像素点的3D-Warping,保证了映射的准确性。3)针对两种不同的3D-Warping方式,分别提出了相应的插值算法。在水平方向上,改进的像素插值算法对紧邻插值和Splatting(散射)插值算法进行了折中,只在映射像素点与待插值像素点很近的情况下才进行紧邻插值,否则进行Splatting插值;在深度方向上,它对Z-Buffer(深度缓存)技术进行了改进,舍弃了与前景物体太远的映射像素点,而对其他映射像素点按深度值进行加权操作。结果实验结果表明,与标准绘制方案的整像素精度相比,绘制时间平均节省了72.05%;与标准绘制方案的半像素精度相比,PSNR平均提高了0.355 d B,SSIM平均提高了0.001 15。结论改进算法非常适用于水平设置相机系统的DIBR技术中的整像素精度绘制,对包含大量深度平坦区域的视频序列效果明显,不但能够提高绘制的速度,而且可以有效地改善绘制的客观质量。