Image Inpainting Technique Incorporating Edge Prior and Attention Mechanism

Recently,deep learning-based image inpainting methods have made great strides in reconstructing damaged regions.However,these methods often struggle to produce satisfactory results when dealing with missing images with large holes,leading to distortions in the structure and blurring of textures.To address these problems,we combine the advantages of transformers and convolutions to propose an image inpainting method that incorporates edge priors and attention mechanisms.The proposed method aims to improve the results of inpainting large holes in images by enhancing the accuracy of structure restoration and the ability to recover texture details.This method divides the inpainting task into two phases:edge prediction and image inpainting.Specifically,in the edge prediction phase,a transformer architecture is designed to combine axial attention with standard self-attention.This design enhances the extraction capability of global structural features and location awareness.It also balances the complexity of self-attention operations,resulting in accurate prediction of the edge structure in the defective region.In the image inpainting phase,a multi-scale fusion attention module is introduced.This module makes full use of multi-level distant features and enhances local pixel continuity,thereby significantly improving the quality of image inpainting.To evaluate the performance of our method.comparative experiments are conducted on several datasets,including CelebA,Places2,and Facade.Quantitative experiments show that our method outperforms the other mainstream methods.Specifically,it improves Peak Signal-to-Noise Ratio(PSNR)and Structure Similarity Index Measure(SSIM)by 1.141~3.234 db and 0.083~0.235,respectively.Moreover,it reduces Learning Perceptual Image Patch Similarity(LPIPS)and Mean Absolute Error(MAE)by 0.0347~0.1753 and 0.0104~0.0402,respectively.Qualitative experiments reveal that our method excels at reconstructing images with complete structural information and clear texture details.Furthermore,our model exhibits impressive performance in terms of the number of parameters,memory cost,and testing time.

Spatial and Contextual Path Network for Image Inpainting

Image inpainting is a kind of use known area of information technology to repair the loss or damage to the area.Image feature extraction is the core of image restoration.Getting enough space for information and a larger receptive field is very important to realize high-precision image inpainting.However,in the process of feature extraction,it is difficult to meet the two requirements of obtaining sufficient spatial information and large receptive fields at the same time.In order to obtain more spatial information and a larger receptive field at the same time,we put forward a kind of image restoration based on space path and context path network.For the space path,we stack three convolution layers for 1/8 of the figure,the figure retained the rich spatial details.For the context path,we use the global average pooling layer,where the accept field is the maximum of the backbone network,and the pooling module can provide global context information for the maximum accept field.In order to better integrate the features extracted from the spatial and contextual paths,we study the fusion module of the two paths.Features fusionmodule first path output of the space and context path,and then through themass normalization to balance the scale of the characteristics,finally the characteristics of the pool will be connected into a feature vector and calculate the weight vector.Features of images in order to extract context information,we add attention to the context path refinement module.Attention modules respectively from channel dimension and space dimension to weighted images,in order to obtain more effective information.Experiments show that our method is better than the existing technology in the quality and quantity of themethod,and further to expand our network to other inpainting networks,in order to achieve consistent performance improvements.

Multi-Layer Deep Sparse Representation for Biological Slice Image Inpainting

Biological slices are an effective tool for studying the physiological structure and evolutionmechanism of biological systems.However,due to the complexity of preparation technology and the presence of many uncontrollable factors during the preparation processing,leads to problems such as difficulty in preparing slice images and breakage of slice images.Therefore,we proposed a biological slice image small-scale corruption inpainting algorithm with interpretability based on multi-layer deep sparse representation,achieving the high-fidelity reconstruction of slice images.We further discussed the relationship between deep convolutional neural networks and sparse representation,ensuring the high-fidelity characteristic of the algorithm first.A novel deep wavelet dictionary is proposed that can better obtain image prior and possess learnable feature.And multi-layer deep sparse representation is used to implement dictionary learning,acquiring better signal expression.Compared with methods such as NLABH,Shearlet,Partial Differential Equation(PDE),K-Singular Value Decomposition(K-SVD),Convolutional Sparse Coding,and Deep Image Prior,the proposed algorithm has better subjective reconstruction and objective evaluation with small-scale image data,which realized high-fidelity inpainting,under the condition of small-scale image data.And theOn2-level time complexitymakes the proposed algorithm practical.The proposed algorithm can be effectively extended to other cross-sectional image inpainting problems,such as magnetic resonance images,and computed tomography images.

图像Inpainting技术原理及在包装印刷图像处理中的应用

在包装印刷行业图像修复问题需要有经验的技术人员进行复杂的手工处理,随着计算机图像处理领域对图像自动处理技术的讨论,INPAINTING技术对包装印刷图像处理提供了新的方法和方向。主要介绍了图像自动修复技术的原理、发展,以及在包装印刷行业的应用。

RESEARCH ON WEIGHTED PRIORITY OF EXEMPLAR-BASED IMAGE INPAINTING

The priority of the filled patch play a key role in the exemplar-based image inpainting, and it should be determined firstly to optimize the process of image inpainting. A modified image inpainting algorithm is proposed by weighted-priority based on the Criminisi algorithm. The improved algorithm demonstrates better relationship between the data term and the confidence term for the optimization of the priority than the classical Criminisi algorithm. By comparing the effect of the inpainted images with different structure, conclusion can be drawn that the optimal priority should be chosen properly for different images with different structures.

INPAINTING ALGORITHM FOR KINECT DEPTH MAP BASED ON FOREGROUND SEGMENTATION

The depth information of the scene indicates the distance between the object and the camera,and depth extraction is a key technology in 3D video system.The emergence of Kinect makes the high resolution depth map capturing possible.However,the depth map captured by Kinect can not be directly used due to the existing holes and noises,which needs to be repaired.We propose a texture combined inpainting algorithm in this paper.Firstly,the foreground is segmented combined with the color characteristics of the texture image to repair the foreground of the depth map.Secondly,region growing is used to determine the match region of the hole in the depth map,and to accurately position the match region according to the texture information.Then the match region is weighted to fill the hole.Finally,a Gaussian filter is used to remove the noise in the depth map.Experimental results show that the proposed method can effectively repair the holes existing in the original depth map and get an accurate and smooth depth map,which can be used to render a virtual image with good quality.

Color Texture Image Inpainting Using the Non Local CTV Model

The classical TV (Total Variation) model has been applied to gray texture image denoising and inpainting previously based on the non local operators, but such model can not be directly used to color texture image inpainting due to coupling of different image layers in color images. In order to solve the inpainting problem for color texture images effectively, we propose a non local CTV (Color Total Variation) model. Technically, the proposed model is an extension of local TV model for gray images but we take account of the coupling of different layers in color images and make use of concepts of the non-local operators. As the coupling of different layers for color images in the proposed model will in-crease computational complexity, we also design a fast Split Bregman algorithm. Finally, some numerical experiments are conducted to validate the performance of the proposed model and its algorithm.

UFC-Net with Fully-Connected Layers and Hadamard Identity Skip Connection for Image Inpainting

Image inpainting is an interesting technique in computer vision and artificial intelligence for plausibly filling in blank areas of an image by referring to their surrounding areas.Although its performance has been improved significantly using diverse convolutional neural network(CNN)-based models,these models have difficulty filling in some erased areas due to the kernel size of the CNN.If the kernel size is too narrow for the blank area,the models cannot consider the entire surrounding area,only partial areas or none at all.This issue leads to typical problems of inpainting,such as pixel reconstruction failure and unintended filling.To alleviate this,in this paper,we propose a novel inpainting model called UFC-net that reinforces two components in U-net.The first component is the latent networks in the middle of U-net to consider the entire surrounding area.The second component is the Hadamard identity skip connection to improve the attention of the inpainting model on the blank areas and reduce computational cost.We performed extensive comparisons with other inpainting models using the Places2 dataset to evaluate the effectiveness of the proposed scheme.We report some of the results.

A 360-Degree Panoramic Image Inpainting Network Using a Cube Map

Inpainting has been continuously studied in the field of computer vision.As artificial intelligence technology developed,deep learning technology was introduced in inpainting research,helping to improve performance.Currently,the input target of an inpainting algorithm using deep learning has been studied from a single image to a video.However,deep learning-based inpainting technology for panoramic images has not been actively studied.We propose a 360-degree panoramic image inpainting method using generative adversarial networks(GANs).The proposed network inputs a 360-degree equirectangular format panoramic image converts it into a cube map format,which has relatively little distortion and uses it as a training network.Since the cube map format is used,the correlation of the six sides of the cube map should be considered.Therefore,all faces of the cube map are used as input for the whole discriminative network,and each face of the cube map is used as input for the slice discriminative network to determine the authenticity of the generated image.The proposed network performed qualitatively better than existing single-image inpainting algorithms and baseline algorithms.

Image inpainting using complex 2-D dual-tree wavelet transform

The dual-tree complex wavelet transform is a useful tool in signal and image process- ing. In this paper, we propose a dual-tree complex wavelet transform （CWT） based algorithm for image inpalnting problem. Our approach is based on Cai, Chan, Shen and Shen＇s framelet-based algorithm. The complex wavelet transform outperforms the standard real wavelet transform in the sense of shift-invariance, directionality and anti-aliasing. Numerical results illustrate the good performance of our algorithm.

An Efficient Video Inpainting Approach Using Deep Belief Network

The video inpainting process helps in several video editing and restoration processes like unwanted object removal,scratch or damage rebuilding,and retargeting.It intends to fill spatio-temporal holes with reasonable content in the video.Inspite of the recent advancements of deep learning for image inpainting,it is challenging to outspread the techniques into the videos owing to the extra time dimensions.In this view,this paper presents an efficient video inpainting approach using beetle antenna search with deep belief network(VIA-BASDBN).The proposed VIA-BASDBN technique initially converts the videos into a set of frames and they are again split into a region of 5*5 blocks.In addition,the VIABASDBN technique involves the design of optimal DBN model,which receives input features from Local Binary Patterns(LBP)to categorize the blocks into smooth or structured regions.Furthermore,the weight vectors of the DBN model are optimally chosen by the use of BAS technique.Finally,the inpainting of the smooth and structured regions takes place using the mean and patch matching approaches respectively.The patch matching process depends upon the minimal Euclidean distance among the extracted SIFT features of the actual and references patches.In order to examine the effective outcome of the VIA-BASDBN technique,a series of simulations take place and the results denoted the promising performance.

Image Inpainting Detection Based on High-Pass Filter Attention Network

Image inpainting based on deep learning has been greatly improved.The original purpose of image inpainting was to repair some broken photos, suchas inpainting artifacts. However, it may also be used for malicious operations,such as destroying evidence. Therefore, detection and localization of imageinpainting operations are essential. Recent research shows that high-pass filteringfull convolutional network (HPFCN) is applied to image inpainting detection andachieves good results. However, those methods did not consider the spatial location and channel information of the feature map. To solve these shortcomings, weintroduce the squeezed excitation blocks (SE) and propose a high-pass filter attention full convolutional network (HPACN). In feature extraction, we apply concurrent spatial and channel attention (scSE) to enhance feature extraction and obtainmore information. Channel attention (cSE) is introduced in upsampling toenhance detection and localization. The experimental results show that the proposed method can achieve improvement on ImageNet.

Weighted Variational Minimization Model for Wavelet Domain Inpainting with Primal-Dual Method

To preserve the edges and details of the image,a new variational model for wavelet domain inpainting was proposed which contained a non-convex regularizer. The non-convex regularizer can utilize the local information of image and perform better than those usual convex ones. In addition, to solve the non-convex minimization problem,an iterative reweighted method and a primaldual method were designed. The numerical experiments show that the new model not only gets better visual effects but also obtains higher signal to noise ratio than the recent method.

Deep learning-based inpainting of saturation artifacts in optical coherence tomography images

Limited by the dynamic range of the detector,saturation artifacts usually occur in optical coherence tomography(OCT)imaging for high scattering media.The available methods are difficult to remove saturation artifacts and restore texture completely in OCT images.We proposed a deep learning-based inpainting method of saturation artifacts in this paper.The generation mechanism of saturation artifacts was analyzed,and experimental and simulated datasets were built based on the mechanism.Enhanced super-resolution generative adversarial networks were trained by the clear–saturated phantom image pairs.The perfect reconstructed results of experimental zebrafish and thyroid OCT images proved its feasibility,strong generalization,and robustness.

Pyramid-VAE-GAN:Transferring hierarchical latent variables for image inpainting

Significant progress has been made in image inpainting methods in recent years.However,they are incapable of producing inpainting results with reasonable structures,rich detail,and sharpness at the same time.In this paper,we propose the Pyramid-VAE-GAN network for image inpainting to address this limitation.Our network is built on a variational autoencoder(VAE)backbone that encodes high-level latent variables to represent complicated high-dimensional prior distributions of images.The prior assists in reconstructing reasonable structures when inpainting.We also adopt a pyramid structure in our model to maintain rich detail in low-level latent variables.To avoid the usual incompatibility of requiring both reasonable structures and rich detail,we propose a novel cross-layer latent variable transfer module.This transfers information about long-range structures contained in high-level latent variables to low-level latent variables representing more detailed information.We further use adversarial training to select the most reasonable results and to improve the sharpness of the images.Extensive experimental results on multiple datasets demonstrate the superiority of our method.Our code is available at https://github.com/thy960112/Pyramid-VAE-GAN.

MULTIGRID METHOD FOR A MODIFIED CURVATURE DRIVEN DIFFUSION MODEL FOR IMAGE INPAINTING

Digital inpainting is a fundamental problem in image processing and many variational models for this problem have appeared recently in the literature. Among them are the very successfully Total Variation （TV） model [11] designed for local inpainting and its improved version for large scale inpainting： the Curvature-Driven Diffusion （CDD） model [10]. For the above two models, their associated Euler Lagrange equations are highly nonlinear partial differential equations. For the TV model there exists a relatively fast and easy to implement fixed point method, so adapting the multigrid method of [24] to here is immediate. For the CDD model however, so far only the well known but usually very slow explicit time marching method has been reported and we explain why the implementation of a fixed point method for the CDD model is not straightforward. Consequently the multigrid method as in [Savage and Chen, Int. J. Comput. Math., 82 （2005）, pp. 1001-1015] will not work here. This fact represents a strong limitation to the range of applications of this model since usually fast solutions are expected. In this paper, we introduce a modification designed to enable a fixed point method to work and to preserve the features of the original CDD model. As a result, a fast and efficient multigrid method is developed for the modified model. Numerical experiments are presented to show the very good performance of the fast algorithm.

Parallel Algorithm and Software for Image Inpainting via Sub-Riemannian Minimizers on the Group of Rototranslations

The paper is devoted to an approach for image inpainting developed on the basis of neurogeometry of vision and sub-Riemannian geometry.Inpainting is realized by completing damaged isophotes(level lines of brightness)by optimal curves for the left-invariant sub-Riemannian problem on the group of rototranslations(motions)of a plane SE(2).The approach is considered as anthropomorphic inpainting since these curves satisfy the variational principle discovered by neurogeometry of vision.A parallel algorithm and software to restore monochrome binary or halftone images represented as series of isophotes were developed.The approach and the algorithm for computation of completing arcs are presented in detail.

Image Inpainting Based on Structural Tensor Edge Intensity Model

In the exemplar-based image inpainting approach,there are usually two major problems:the unreasonable calculation of priority and only considering the color features in the patch lookup strategy.In this paper,we propose an image inpainting approach based on the structural tensor edge intensity model.First,we use the progressive scanning inpainting method to avoid the image filling order being affected by the priority function.Then,we use the edge intensity model to build the patches similarity function for correctly identifying the local image structure.Finally,the balance operator is used to restrict the excessive propagation of structural information to ensure the correct structural reconstruction.The experimental results show that the our approach is comparable and even superior to some state-of-the-art inpainting algorithms.

Image decomposing for inpainting using compressed sensing in DCT domain

Inpainting images with occlusion or corruption is a challenging task. Most existing algorithms are pixel based, which construct a statistical model from image features. However, in these algorithms, the frequency component is not sufficiently addressed. In this paper, we propose a novel algorithm that utilizes compressed sensing （CS） in frequency domain to reconstruct corrupted images. In order to reconstruct image, we first decompose the image into two functions with different basic characteristics - structure component and textual component. We seek a sparse representation for the functions and use the DCT coefficients of this representation to generate an over-complete dictionary. Experimental results on real world datasets demonstrate the efficacy of our method in image inpainting. We compare our method with three state-of-the-art inpalnting algorithms and demonstrate its advantages in terms of both quantitative and qualitative aspects.