联合语义分割与边缘重建的深度学习图像修复

Deep learning image inpainting combining semantic segmentation reconstruction and edge reconstruction

目的传统图像修复方法缺乏对图像高级语义的理解,只能应对结构纹理简单的小面积受损。现有的端到端深度学习图像修复方法在大量训练图像的支持下克服了上述局限性,但由于这些方法试图在约束不足的情况下恢复整个目标,修复的图像往往存在边界模糊和结构扭曲问题。对此,本文提出一种语义分割结构与边缘结构联合指导的深度学习图像修复方法。方法该方法将图像修复任务分解为语义分割重建、边缘重建和内容补全3个阶段。首先重建缺失区域的语义分割结构,然后利用重建的语义分割结构指导缺失区域边缘结构的重建,最后利用重建的语义分割结构与边缘结构联合指导图像缺失区域内容的补全。结果在CelebAMask-HQ(celebfaces attributes mask high quality)人脸数据集和Cityscapes城市景观数据集上,将本文方法与其他先进的图像修复方法进行对比实验。在掩膜比例为50%~60%的情况下,与性能第2的方法相比,本文方法在Celebamask-HQ数据集上的平均绝对误差降低了4.5%,峰值信噪比提高了1.6%,结构相似性提高了1.7%;在Cityscapes数据集上平均绝对误差降低了4.2%,峰值信噪比提高了1.5%,结构相似性提高了1.9%。结果表明,本文方法在平均绝对误差、峰值信噪比和结构相似性3个指标上均优于对比方法,且生成的图像边界清晰,视觉上更加合理。结论本文提出的3阶段图像修复方法在语义分割结构与边缘结构的联合指导下,有效减少了结构重建错误。当修复涉及大面积缺失时,该方法比现有方法具有更高的修复质量。

Objective Image in-painting is to reconstruct the missing areas of distorted images.This technique is widely used in multiple scenes like image editing,image de-noising,cultural relics preservation.Conventional image in-painting methods are based on patch blocks to fill the missing pixels or to transmit the pixels to the missing region via diffusion mechanism.These methods have achieved regular effects or small defects in in-painting.However,due to the lack of semantic understanding of the image,more generated images often have a non-photorealistic sense of inconsistent semantic structure when filling large-scale consistent holes.Deep learning-based in-painting method can learn the high-level semantic information of the image from a large number of data.Although these methods have made significant progress in image inpainting,they are often unable to reconstruct feasible structures.Current methods are focused on target-completed restoration without sufficient constraints,so the generated images often have the problems of fuzzy boundaries and distorted structures.Method Our research is aimed to develop a deep image inpainting method guided by semantic segmentation and edge.It divides the image inpainting task into three steps:1)semantic segmentation reconstruction,2)edge reconstruction and 3)content restoration.First,the semantic segmentation reconstruction module reconstructs the semantic segmentation structure.Then,the reconstructed semantic segmentation structure is used to guide the reconstruction of the edge structure of the missing area.Finally,the reconstructed semantic segmentation structure and edge structure are used to guide the content restoration of the missing area.Semantic segmentation can represent the global structure information of the image well.1)The reconstruction of the semantic segmentation structure can improve the accuracy of edge structure-reconstructed.2)Edge contains rich structural information,reconstructing the edge structure is beneficial to generate more inner details of object.3)Under the guidance of reconstructed semantic segmentation structure and edge structure,the content restoration can use texture in-painting to clear the boundary of the generated image.The structure is more reasonable,and the texture is more real.Our network structure is based on the generative adversarial network(GAN-based),including generator and discriminator.The generator network uses encoder-decoder structure and the discriminator network uses 70×70 PatchGAN structure.Joint loss is adopted in terms of loss function in the three steps,which can approach the in-painting results of each step to real results.The two reconstructed modules of semantic segmentation and edge use adversarial loss and feature matching loss.Our feature matching loss used actually includes L1 loss function.Feature matching loss is similar to perceptual loss,which can clarify the ground truth issue of semantic segmentation structure and edge structure.The content restoration module can add the perception loss and style loss in the context of image in-painting when style loss can reduce the“checkerboard”artifact caused by transpose convolution layer.Result First,we analyze the performance of semantic segmentation reconstruction module quantitatively and qualitatively.The results show that the semantic segmentation reconstruction module can reconstruct the feasibility of semantic segmentation structure.When the mask is small,the pixel accuracy can reach 99.16%,and for the larger mask,the pixel accuracy can also reach 92.64%.Next,we compare the edge reconstruction results quantitatively.It shows that the accuracy and recall of the reconstructed edge structure are optimized further under the guidance of the semantic segmentation structure.Finally,the method proposed is compared with four popular image in-painting methods on CelebAMask HQ(celebfaces attributes mask high quality)dataset and Cityscapes dataset.When the mask ratio is 50%~60%,compared with the second-performing method,the mean absolute error(MAE)on the CelebAMask-HQ dataset is reduced by 4.5%,the peak signal-to-noise ratio(PSNR)is increased by 1.6%,and the structure similarity index measure(SSIM)is increased by 1.7%;the MAE on the Cityscapes dataset is reduced by 4.2%,the PSNR is increased by 1.5%,and the SSIM is increased by 1.9%.Our method is optimized for the three indexes of MAE,PSNR and SSIM,the generated image has more clear boundaries and visibility.Conclusion Our 3-steps image in-painting method introduces the guidance of semantic segmentation structure,which can significantly improve the accuracy of edge reconstruction.In addition,it can reduce structure reconstruction errors effectively through the joint guidance of semantic segmentation structure and edge structure.It has stronger potentials in-painting quality for large-area deletions-oriented in-painting task.