基于残差卷积注意力网络的视频修复

Video inpainting based on residual convolution attention network

视频修复旨在填补视频中的缺失区域,由于很难精确保持修复内容的时空一致性,故视频修复仍具有挑战性。针对现有视频修复中存在的修复结果语义信息不连续,出现视频模糊和时间伪影,以及网络设计越来越复杂,网络整体速度变慢的问题,本文提出了一种基于残差网络的卷积注意力网络(RCAN)用以视频修复。通过将自注意力机制和全局注意力机制引入进残差网络,增强网络对所有输入帧的时空特征的学习能力,并采用时空对抗损失函数进行优化,提高视频修复的质量。同时网络还能够高度自由地定义层数和参数量,提高网络的实际应用能力。实验结果表明,该网络在DAVIS和YouTube-VOS数据集上取得了PSNR为30.68 dB,SSIM为0.961,FID为0.113的平均修复结果,基本符合实际场景对模型的修复质量要求,为视频修复提供了一种新思路。

Video inpainting,which aims at filling in missing regions of a video,remains challenging due to the difficulty of preserving the precise spatial and temporal coherence of video contents.In order to solve the problems of discontinuous semantic information,video blurriness and temporal artifact,and more and more complex network design,the overall speed of the network becoming slow,this paper proposes a residual convolution attention network(RCAN)for video inpainting.By introducing the self-attention mechanism and the global attention mechanism into the residual network,the ability of the network to learn the spatio-temporal features of all input frames is enhanced.This method proposes a spatial-temporal adversarial loss function to optimize RCAN,which improves the quality of video inpainting.At the same time,the network can define the number of layers and parameters with a high degree of freedom to improve the practical application ability of the network.Experimental results show that the network can achieve an average inpainting result in that the PSNR is 30.68 dB,the SSIM is 0.961,and the FID is 0.113 on DAVIS and YouTube-VOS data sets.This method meets the inpainting quality requirements of the actual scene on the model and provides a new idea for video inpainting.