联合视口预测和超分辨率重建的实时VR全景视频流式传输方法

Live VR Panoramic Video Streaming Method Combing Viewport Prediction and Super-Resolution

现有基于视口预测的VR全景视频流式传输方法尚未有效地考虑VR视频的时间维度信息,并且忽略了客户端在提高视频重建质量上应发挥的作用.为了进一步提高传输系统的综合性能及用户体验质量,提出一种联合视口预测和超分辨率重建的实时VR全景视频流式传输方法.在服务器端提出时间非局部注意力模块,并将其嵌入GhostNet中对VR全景视频进行全局上下文建模,捕获视频内容特征在时间维度上的长距离依赖;在客户端提出轻量级VR全景视频超分辨率重建模型,对来自服务器端预测视口内的次要内容进行质量增强及投影失真优化.在VR全景视频用户头部运动数据集上的实验结果表明,所提方法的平均视口预测精度和平均带宽占用分别为95.6%和52.9%,与5种代表性的传输方法相比,该方法能够获得更高的视口预测精度和更低的带宽占用,同时具有良好的视频重建质量和较低的计算资源消耗.

The existing VR panoramic video streaming methods based on viewport prediction have not effec-tively considered the temporal dimension information of VR video,and ignored the role that the client end should play in improving the video reconstruction quality.To further improve the comprehensive perform-ance of the streaming system and the user quality of experience,a live VR panoramic video streaming method that combines viewport prediction and super-resolution reconstruction is proposed.At the server end,the method captures the long-distance dependency in the temporal dimension of video content features by embedding the proposed temporal non-local attention module TNAM into GhostNet to model the global context of VR panoramic video;At the client end,the proposed lightweight VR panoramic video su-per-resolution reconstruction model LVRSR is used to enhance the quality and optimize the projection dis-tortion of the secondary content within the predicted viewport from the server end.The experimental results on the VR panoramic video user head mo-tion dataset show that the average viewport prediction accuracy and average bandwidth usage of the method are 95.6%and 52.9%,respectively.Compared with five repre-sentative streaming methods,the method can achieve higher viewport prediction accuracy and lower band-width usage,while having good video reconstruction quality and low computational resource consumption.