基于网格流的视频修补网络

MOVIE:Mesh oriented video inpainting network

视频修补的目的是基于视频帧之间的时空域上下文信息修补空洞.现有的方法由于不能准确地对运动轨迹进行建模经常导致修补结果无法保持良好的时空一致性.为此引入灵活的形状自适应网格作为基本处理单元,将网格流用于运动表示,提出了一个基于网格流的视频修补网络,通过先预测网络流再添补空洞区域的方式对视频进行修补.具体地,首先设计了一个网格流预测模块用于预测视频中可见内容的网格流的预测和一个网格流修补模块用于修补视频中空洞区域的网格流,通过这种方式将网格流的预测和修补解耦以达到更容易训练优化的目的.我们进一步设计了一个混合损失函数用于同时优化可见区域、修补区域和整个视频帧范围的网格流预测结果.为修正经过网格流变换引起的失真现象,最后设计了一个修补优化网络.大量试验结果证明,本文提出的方法不仅从主观评判和客观指标得到相比于现有方法更好的修补结果,而且相比于现有最快的方法达到了4倍的速度提升.

Video inpainting aims to fill the holes across different frames upon limited spatio-temporal contexts.The existing schemes still suffer from achieving precise spatio-temporal coherence especially in hole areas due to inaccurate modeling of motion trajectories.In this paper,we introduce fexible shape-adaptive mesh as basic processing unit and mesh flow as motion representation,which has the capability of describing complex motions in hole areas more precisely and efficiently.We propose a Mesh Oriented Video Inpainting nEtwork,dubbed MOVIE,to estimate mesh flows then complete the hole region in the video.Specifically,we first design a mesh flow estimation module and a mesh flow completion module to estimate the mesh flow for visible contents and holes in a sequential way,which decouples the mesh flow estimation for visible and corrupted contents for easy optimization.A hybrid loss function is further introduced to optimize the flow estimation performance for the visible regions,the entire frames and the inpainted regions respectively.Then we design a polishing network to correct the distortion of the inpainted results caused by mesh flow transformation.Extensive experiments show that MOVIE not only achieves over four-times speed-up in completing the missing area,but also yields more promising results with much better inpainting quality in both quantitative and perceptual metrics.