基于多注意力机制的纹理感知视频修复方法

Texture-aware video inpainting algorithm based on the multi-attention mechanism

针对现有视频修复方法无法有效利用远处空间内容信息而导致修复结果中存在结构和纹理不合理的问题,提出了一种基于多注意力机制的纹理感知视频修复方法。该方法设计了由多头时空注意力和单图局部注意力构成的多注意力机制以保证全局结构并增强局部纹理,其中多头时空注意力关注整体时空信息,单图局部注意力通过局部窗口的自注意力机制精炼提取局部信息。另外,采用可即插即用的快速傅里叶卷积层残差块代替前馈网络中的普通卷积,将感受野扩展为整个图像,进一步增强了模型对图像纹理和结构的全局信息的获取能力。快速傅里叶卷积层残差块和单图局部注意力相辅相成,共同提升局部纹理的修复质量。在YouTube-VOS和DAVIS数据集上的实验结果表明,虽然提出的方法修复结果的客观质量评价仅次于最优方法Fuseformer,但其参数量和运行时间分别下降了54.8%和21.5%,而且能够生成视觉上更逼真、语义上更合理的修复内容。

Existing video inpainting methods cannot effectively utilize distant spatial contents,which results in unreasonable structures and textures.To solve this problem,a texture-aware video inpainting algorithm based on the multi-attention mechanism is proposed in this paper.The algorithm designs a multi-attention mechanism composed of multi-head spatiotemporal attention and single-image local attention,guaranteeing global structures and enriching local textures.Multi-head spatial-temporal attention focuses on the overall spatial-temporal information,and single-image local attention distills local information through local windows of the self-attention mechanism.A plug-and-play fast Fourier convolution layer residual block is used to replace vanilla convolution in feedforward networks,expanding the receptive field into the entire image so that the global structure and texture of a single frame image can be enriched.The fast Fourier convolutional layer residual block and the single-image local attention complement each other and jointly promote the quality of local textures.Experimental results on YouTube-VOS and DAVIS datasets show that although the proposed method ranks second only to the optimal method Fuseformer on objective metrics,the number of parameters and running time are reduced by 54.8%and 21.5%respectively.And the proposed method can generate more visually realistic and semantically reasonable contents.