基于3D-ResNet双流网络的VR病评估模型

VR sickness estimation model based on 3D-ResNet two-stream network

为了准确地评估VR视频引起不适的程度,提出基于3D双流卷积神经网络的VR病评估模型.模仿人类视觉系统的2条通路,建立外观流和运动流2个子网络;将2D-ResNet50模型改为3D模型,增加一个深度通道,用以学习视频中的时序信息.加入3D-CBAM注意力模块提高了各帧通道之间的空间关联,增强关键信息,去除冗余信息.采用后端融合的方法,实现2个子网络结果的融合.在公开视频数据集上进行实验验证,结果表明,通过3D-CBAM注意力模块引入注意力机制,使得外观流和运动流网络的VR病评估精度分别提升了1.7%和3.6%,与现有文献相比,融合的双流网络模型的精度得到了较大的提升,精度达到93.7%.

A VR sickness estimation method was proposed based on 3D two-stream convolutional neural network in order to accurately estimate VR sickness of VR video.Two sub-networks,which were appearance flow and motion flow,were constructed to mimic the two pathways of human visual system.2D-ResNet50 model was changed to 3D model and a depth channel was added to learn the timing information in videos.3D-CBAM attention module was introduced to improve the spatial correlation between channels of each frame.Then the key information was enhanced and redundant information was suppressed.The back-end fusion method was used to fuse the results of the two sub-networks.Experiments were conducted on a public video dataset.The experimental results showed that the accuracy of the appearance stream network and the motion stream network was improved by 1.7% and 3.6%respectively by introducing the attention mechanism.The accuracy of the fused two-stream network was improved to93.7%,which outperformed other literatures.