Image Inpainting Technique Incorporating Edge Prior and Attention Mechanism

Recently,deep learning-based image inpainting methods have made great strides in reconstructing damaged regions.However,these methods often struggle to produce satisfactory results when dealing with missing images with large holes,leading to distortions in the structure and blurring of textures.To address these problems,we combine the advantages of transformers and convolutions to propose an image inpainting method that incorporates edge priors and attention mechanisms.The proposed method aims to improve the results of inpainting large holes in images by enhancing the accuracy of structure restoration and the ability to recover texture details.This method divides the inpainting task into two phases:edge prediction and image inpainting.Specifically,in the edge prediction phase,a transformer architecture is designed to combine axial attention with standard self-attention.This design enhances the extraction capability of global structural features and location awareness.It also balances the complexity of self-attention operations,resulting in accurate prediction of the edge structure in the defective region.In the image inpainting phase,a multi-scale fusion attention module is introduced.This module makes full use of multi-level distant features and enhances local pixel continuity,thereby significantly improving the quality of image inpainting.To evaluate the performance of our method.comparative experiments are conducted on several datasets,including CelebA,Places2,and Facade.Quantitative experiments show that our method outperforms the other mainstream methods.Specifically,it improves Peak Signal-to-Noise Ratio(PSNR)and Structure Similarity Index Measure(SSIM)by 1.141~3.234 db and 0.083~0.235,respectively.Moreover,it reduces Learning Perceptual Image Patch Similarity(LPIPS)and Mean Absolute Error(MAE)by 0.0347~0.1753 and 0.0104~0.0402,respectively.Qualitative experiments reveal that our method excels at reconstructing images with complete structural information and clear texture details.Furthermore,our model exhibits impressive performance in terms of the number of parameters,memory cost,and testing time.

Spatial and Contextual Path Network for Image Inpainting

Image inpainting is a kind of use known area of information technology to repair the loss or damage to the area.Image feature extraction is the core of image restoration.Getting enough space for information and a larger receptive field is very important to realize high-precision image inpainting.However,in the process of feature extraction,it is difficult to meet the two requirements of obtaining sufficient spatial information and large receptive fields at the same time.In order to obtain more spatial information and a larger receptive field at the same time,we put forward a kind of image restoration based on space path and context path network.For the space path,we stack three convolution layers for 1/8 of the figure,the figure retained the rich spatial details.For the context path,we use the global average pooling layer,where the accept field is the maximum of the backbone network,and the pooling module can provide global context information for the maximum accept field.In order to better integrate the features extracted from the spatial and contextual paths,we study the fusion module of the two paths.Features fusionmodule first path output of the space and context path,and then through themass normalization to balance the scale of the characteristics,finally the characteristics of the pool will be connected into a feature vector and calculate the weight vector.Features of images in order to extract context information,we add attention to the context path refinement module.Attention modules respectively from channel dimension and space dimension to weighted images,in order to obtain more effective information.Experiments show that our method is better than the existing technology in the quality and quantity of themethod,and further to expand our network to other inpainting networks,in order to achieve consistent performance improvements.

Multi-Layer Deep Sparse Representation for Biological Slice Image Inpainting

Biological slices are an effective tool for studying the physiological structure and evolutionmechanism of biological systems.However,due to the complexity of preparation technology and the presence of many uncontrollable factors during the preparation processing,leads to problems such as difficulty in preparing slice images and breakage of slice images.Therefore,we proposed a biological slice image small-scale corruption inpainting algorithm with interpretability based on multi-layer deep sparse representation,achieving the high-fidelity reconstruction of slice images.We further discussed the relationship between deep convolutional neural networks and sparse representation,ensuring the high-fidelity characteristic of the algorithm first.A novel deep wavelet dictionary is proposed that can better obtain image prior and possess learnable feature.And multi-layer deep sparse representation is used to implement dictionary learning,acquiring better signal expression.Compared with methods such as NLABH,Shearlet,Partial Differential Equation(PDE),K-Singular Value Decomposition(K-SVD),Convolutional Sparse Coding,and Deep Image Prior,the proposed algorithm has better subjective reconstruction and objective evaluation with small-scale image data,which realized high-fidelity inpainting,under the condition of small-scale image data.And theOn2-level time complexitymakes the proposed algorithm practical.The proposed algorithm can be effectively extended to other cross-sectional image inpainting problems,such as magnetic resonance images,and computed tomography images.

RESEARCH ON WEIGHTED PRIORITY OF EXEMPLAR-BASED IMAGE INPAINTING

The priority of the filled patch play a key role in the exemplar-based image inpainting, and it should be determined firstly to optimize the process of image inpainting. A modified image inpainting algorithm is proposed by weighted-priority based on the Criminisi algorithm. The improved algorithm demonstrates better relationship between the data term and the confidence term for the optimization of the priority than the classical Criminisi algorithm. By comparing the effect of the inpainted images with different structure, conclusion can be drawn that the optimal priority should be chosen properly for different images with different structures.

Gait recognition based on Wasserstein generating adversarial image inpainting network

Aiming at the problem of small area human occlusion in gait recognition,a method based on generating adversarial image inpainting network was proposed which can generate a context consistent image for gait occlusion area.In order to reduce the effect of noise on feature extraction,the stacked automatic encoder with robustness was used.In order to improve the ability of gait classification,the sparse coding was used to express and classify the gait features.Experiments results showed the effectiveness of the proposed method in comparison with other state-of-the-art methods on the public databases CASIA-B and TUM-GAID for gait recognition.

A 360-Degree Panoramic Image Inpainting Network Using a Cube Map

Inpainting has been continuously studied in the field of computer vision.As artificial intelligence technology developed,deep learning technology was introduced in inpainting research,helping to improve performance.Currently,the input target of an inpainting algorithm using deep learning has been studied from a single image to a video.However,deep learning-based inpainting technology for panoramic images has not been actively studied.We propose a 360-degree panoramic image inpainting method using generative adversarial networks(GANs).The proposed network inputs a 360-degree equirectangular format panoramic image converts it into a cube map format,which has relatively little distortion and uses it as a training network.Since the cube map format is used,the correlation of the six sides of the cube map should be considered.Therefore,all faces of the cube map are used as input for the whole discriminative network,and each face of the cube map is used as input for the slice discriminative network to determine the authenticity of the generated image.The proposed network performed qualitatively better than existing single-image inpainting algorithms and baseline algorithms.

Layered image inpainting based on image decomposition

We propose a layered image inpainting scheme based on image decomposition. The damaged image is first decomposed into three layers： cartoon, edge, and texture. The cartoon and edge layers are repaired using an adaptive offset operator that can fill-in damaged image blocks while preserving sharpness of edges. The missing information in the texture layer is generated with a texture synthesis method. By using discrete cosine transform （DCT） in image decomposition and trading between resolution and computation complexity in texture synthesis, the processing time is kept at a reasonable level.

UFC-Net with Fully-Connected Layers and Hadamard Identity Skip Connection for Image Inpainting

Image inpainting is an interesting technique in computer vision and artificial intelligence for plausibly filling in blank areas of an image by referring to their surrounding areas.Although its performance has been improved significantly using diverse convolutional neural network(CNN)-based models,these models have difficulty filling in some erased areas due to the kernel size of the CNN.If the kernel size is too narrow for the blank area,the models cannot consider the entire surrounding area,only partial areas or none at all.This issue leads to typical problems of inpainting,such as pixel reconstruction failure and unintended filling.To alleviate this,in this paper,we propose a novel inpainting model called UFC-net that reinforces two components in U-net.The first component is the latent networks in the middle of U-net to consider the entire surrounding area.The second component is the Hadamard identity skip connection to improve the attention of the inpainting model on the blank areas and reduce computational cost.We performed extensive comparisons with other inpainting models using the Places2 dataset to evaluate the effectiveness of the proposed scheme.We report some of the results.

Image inpainting using complex 2-D dual-tree wavelet transform

The dual-tree complex wavelet transform is a useful tool in signal and image process- ing. In this paper, we propose a dual-tree complex wavelet transform （CWT） based algorithm for image inpalnting problem. Our approach is based on Cai, Chan, Shen and Shen＇s framelet-based algorithm. The complex wavelet transform outperforms the standard real wavelet transform in the sense of shift-invariance, directionality and anti-aliasing. Numerical results illustrate the good performance of our algorithm.

Image Inpainting Detection Based on High-Pass Filter Attention Network

Image inpainting based on deep learning has been greatly improved.The original purpose of image inpainting was to repair some broken photos, suchas inpainting artifacts. However, it may also be used for malicious operations,such as destroying evidence. Therefore, detection and localization of imageinpainting operations are essential. Recent research shows that high-pass filteringfull convolutional network (HPFCN) is applied to image inpainting detection andachieves good results. However, those methods did not consider the spatial location and channel information of the feature map. To solve these shortcomings, weintroduce the squeezed excitation blocks (SE) and propose a high-pass filter attention full convolutional network (HPACN). In feature extraction, we apply concurrent spatial and channel attention (scSE) to enhance feature extraction and obtainmore information. Channel attention (cSE) is introduced in upsampling toenhance detection and localization. The experimental results show that the proposed method can achieve improvement on ImageNet.

ORTHOGONAL-DIRECTIONAL FORWARD DIFFUSION IMAGE INPAINTING AND DENOISING MODEL

In this paper,an orthogonal-directional forward diffusion Partial Differential Equation(PDE) image inpainting and denoising model which processes image based on variation problem is proposed.The novel model restores the damaged information and smoothes the noise in image si-multaneously.The model is morphological invariant which processes image based on the geometrical property.The regularization item of it diffuses along and cross the isophote,and then the known image information is transported into the target region through two orthogonal directions.The cross isophote diffusion part is the TV(Total Variation) equation and the along isophote diffusion part is the inviscid Helmholtz vorticity equation.The equivalence between the Helmholtz equation and the inpainting PDEs is proved.The model with the fidelity item which is used in the whole image domain denoises while preserving edges.So the novel model could inpaint and denoise simultaneously.Both theoretical analysis and experiments have verified the validity of the novel model proposed in this paper.

Deep learning-based inpainting of saturation artifacts in optical coherence tomography images

Limited by the dynamic range of the detector,saturation artifacts usually occur in optical coherence tomography(OCT)imaging for high scattering media.The available methods are difficult to remove saturation artifacts and restore texture completely in OCT images.We proposed a deep learning-based inpainting method of saturation artifacts in this paper.The generation mechanism of saturation artifacts was analyzed,and experimental and simulated datasets were built based on the mechanism.Enhanced super-resolution generative adversarial networks were trained by the clear–saturated phantom image pairs.The perfect reconstructed results of experimental zebrafish and thyroid OCT images proved its feasibility,strong generalization,and robustness.

SemID: Blind Image Inpainting with Semantic Inconsistency Detection

Most existing image inpainting methods aim to fill in the missing content in the inside-hole region of the target image. However, the areas to be restored in realistically degraded images are unspecified. Previous studies have failed to recover the degradations due to the absence of the explicit mask indication. Meanwhile, inconsistent patterns are blended complexly with the image content. Therefore, estimating whether certain pixels are out of distribution and considering whether the object is consistent with the context is necessary. Motivated by these observations, a two-stage blind image inpainting network, which utilizes global semantic features of the image to locate semantically inconsistent regions and then generates reasonable content in the areas, is proposed. Specifically, the representation differences between inconsistent and available content are first amplified, iteratively predicting the region to be restored from coarse to fine. A confidence-driven inpainting network based on prediction masks is then used to estimate the information regarding missing regions. Furthermore, a multiscale contextual aggregation module is introduced for spatial feature transfer to refine the generated contents. Extensive experiments over multiple datasets demonstrate that the proposed method can generate visually plausible and structurally complete results that are particularly effective in recovering diverse degraded images.

MILG:Realistic lip-sync video generation with audio-modulated image inpainting

Existing lip synchronization(lip-sync)methods generate accurately synchronized mouths and faces in a generated video.However,they still confront the problem of artifacts in regions of non-interest(RONI),e.g.,background and other parts of a face,which decreases the overall visual quality.To solve these problems,we innovatively introduce diverse image inpainting to lip-sync generation.We propose Modulated Inpainting Lip-sync GAN(MILG),an audio-constraint inpainting network to predict synchronous mouths.MILG utilizes prior knowledge of RONI and audio sequences to predict lip shape instead of image generation,which can keep the RONI consistent.Specifically,we integrate modulated spatially probabilistic diversity normalization(MSPD Norm)in our inpainting network,which helps the network generate fine-grained diverse mouth movements guided by the continuous audio features.Furthermore,to lower the training overhead,we modify the contrastive loss in lipsync to support small-batch-size and few-sample training.Extensive experiments demonstrate that our approach outperforms the existing state-of-the-art of image quality and authenticity while keeping lip-sync.

Pyramid-VAE-GAN:Transferring hierarchical latent variables for image inpainting

Significant progress has been made in image inpainting methods in recent years.However,they are incapable of producing inpainting results with reasonable structures,rich detail,and sharpness at the same time.In this paper,we propose the Pyramid-VAE-GAN network for image inpainting to address this limitation.Our network is built on a variational autoencoder(VAE)backbone that encodes high-level latent variables to represent complicated high-dimensional prior distributions of images.The prior assists in reconstructing reasonable structures when inpainting.We also adopt a pyramid structure in our model to maintain rich detail in low-level latent variables.To avoid the usual incompatibility of requiring both reasonable structures and rich detail,we propose a novel cross-layer latent variable transfer module.This transfers information about long-range structures contained in high-level latent variables to low-level latent variables representing more detailed information.We further use adversarial training to select the most reasonable results and to improve the sharpness of the images.Extensive experimental results on multiple datasets demonstrate the superiority of our method.Our code is available at https://github.com/thy960112/Pyramid-VAE-GAN.

MULTIGRID METHOD FOR A MODIFIED CURVATURE DRIVEN DIFFUSION MODEL FOR IMAGE INPAINTING

Digital inpainting is a fundamental problem in image processing and many variational models for this problem have appeared recently in the literature. Among them are the very successfully Total Variation （TV） model [11] designed for local inpainting and its improved version for large scale inpainting： the Curvature-Driven Diffusion （CDD） model [10]. For the above two models, their associated Euler Lagrange equations are highly nonlinear partial differential equations. For the TV model there exists a relatively fast and easy to implement fixed point method, so adapting the multigrid method of [24] to here is immediate. For the CDD model however, so far only the well known but usually very slow explicit time marching method has been reported and we explain why the implementation of a fixed point method for the CDD model is not straightforward. Consequently the multigrid method as in [Savage and Chen, Int. J. Comput. Math., 82 （2005）, pp. 1001-1015] will not work here. This fact represents a strong limitation to the range of applications of this model since usually fast solutions are expected. In this paper, we introduce a modification designed to enable a fixed point method to work and to preserve the features of the original CDD model. As a result, a fast and efficient multigrid method is developed for the modified model. Numerical experiments are presented to show the very good performance of the fast algorithm.

Self-adaptively inpainting gaps between electrical image pads according to the image fabric

Images created from measurements made by wireline microresistivity imaging tools have longitudinal gaps when the well circumference exceeds the total width of the pad-mounted electrode arrays.The gap size depends on the tool design and borehole size,and the null data in these gaps negatively aff ect the quantitative evaluation of reservoirs.Images with linear and texture features obtained from microresistivity image logs have distinct dual fabric features because of logging principles and various geological phenomena.Linear image features usually include phenomena such as fractures,bedding,and unconformities.Contrarily,texture-based image features usually indicate phenomena such as vugs and rock matrices.According to the characteristics of this fabric-based binary image structure and guided by the practice of geological interpretation,an adaptive inpainting method for the blank gaps in microresistivity image logs is proposed.For images with linear features,a sinusoidal tracking inpainting algorithm based on an evaluation of the validity and continuity of pixel sets is used.Contrarily,the most similar target transplantation algorithm is applied to texture-based images.The results obtained for measured electrical imaging data showed that the full borehole image obtained by the proposed method,whether it was a linear structural image refl ecting fracture and bedding or texture-based image refl ecting the matrix and pore of rock,had substantially good inpainting quality with enhanced visual connectivity.The proposed method was eff ective for inpainting electrical image logs with large gaps and high angle fractures with high heterogeneity.Moreover,ladder and block artifacts were rare,and the inpainting marks were not obvious.In addition,detailed full borehole images obtained by the proposed method will provide an essential basis for interpreting geological phenomena and reservoir parameters.

Image Inpainting Based on Structural Tensor Edge Intensity Model

In the exemplar-based image inpainting approach,there are usually two major problems:the unreasonable calculation of priority and only considering the color features in the patch lookup strategy.In this paper,we propose an image inpainting approach based on the structural tensor edge intensity model.First,we use the progressive scanning inpainting method to avoid the image filling order being affected by the priority function.Then,we use the edge intensity model to build the patches similarity function for correctly identifying the local image structure.Finally,the balance operator is used to restrict the excessive propagation of structural information to ensure the correct structural reconstruction.The experimental results show that the our approach is comparable and even superior to some state-of-the-art inpainting algorithms.

Parallel Algorithm and Software for Image Inpainting via Sub-Riemannian Minimizers on the Group of Rototranslations

The paper is devoted to an approach for image inpainting developed on the basis of neurogeometry of vision and sub-Riemannian geometry.Inpainting is realized by completing damaged isophotes(level lines of brightness)by optimal curves for the left-invariant sub-Riemannian problem on the group of rototranslations(motions)of a plane SE(2).The approach is considered as anthropomorphic inpainting since these curves satisfy the variational principle discovered by neurogeometry of vision.A parallel algorithm and software to restore monochrome binary or halftone images represented as series of isophotes were developed.The approach and the algorithm for computation of completing arcs are presented in detail.

Analysis of a Hyperbolic-Parabolic Type System for Image Inpainting

In this paper we consider the initial boundary value problem of a hyperbolic-parabolic type system for image inpainting in a 2-D bounded domain, and establish the existence of weak solutions to the system by employing the method of vanishing viscosity.