基于强化学习的立体全景视频自适应流

Adaptive Streaming of Stereoscopic Panoramic Video Based on Reinforcement Learning

针对当前立体全景视频传输缺少有效的流自适应方法,且传统全景视频流自适应策略传输双目立体全景视频使得传输数据加倍,所需带宽巨大的问题,该文提出一种基于多智能体强化学习的立体全景视频非对称传输自适应流方法,以实时应对网络带宽波动。首先,根据人眼对视频显著性区域的偏爱,左右视点中每个瓦片(tile)对立体视频的感知质量的贡献度不同,提出一个基于tiles的左右视点观看概率预测方法。其次,设计了一种基于策略-评价(Actor-Critic)的多智能体强化学习框架,对左右视点进行联合码率控制。最后,根据模型结构和双目抑制原理,设计合理的奖励函数。实验结果表明,与传统流自适应传输策略相比,该文所提方法更加适用于基于tiles的立体全景视频传输,实现在有限带宽下提高用户的体验质量(QoE),为立体全景视频联合码率控制提供了一种全新的方法和思路。

Currently,an effective stream adaptation method for stereo panoramic video transmission is missing.However,the traditional panoramic video adaptive streaming strategy for transmitting binocular stereo panoramic video suffers from the problem of doubling the transmission data and requiring huge bandwidth.A multi-agent reinforcement learning based stereo panoramic video asymmetric transmission adaptive streaming method is proposed in this paper to cope with the limited bandwidth and fluctuation of network bandwidth in real time.First,due to the human eye's preference for the saliency regions of video,each tile in the left and right viewpoints of stereoscopic video contributes differently to the perceptual quality,and a tiles-based method for predicting the watching probability of left and right viewpoint is proposed.Second,a multi-agent reinforcement learning framework based on policy-value(Actor-Critic)is designed for joint rate control of left and right viewpoints.Finally,a reasonable reward function is designed based on the model structure and the principle of binocular suppression.The experimental results show that the proposed method is more suitable for tiles-based stereo panoramic video transmission than the traditional self-adaptive stream transmission strategy.A novel approach is proposed for stereo panoramic video joint rate control and user Quality of Experience(QoE)improvement under limited bandwidth.