A Novel Variational Image Model： Towards a Unified Approach to Image Editing

In this paper we propose a unified variational image editing model. It interprets image editing as a variational problem concerning the adaptive adjustments to the zero- and first-derivatives of the images which correspond to the color and gradient items. By varying the definition domain of each of the two items as well as applying diverse operators, the new model is capable of tackling a variety of image editing tasks. It achieves visually better seamless image cloning effects than existing approaches. It also induces a new and efficient solution to adjusting the color of an image interactively and locally. Other image editing tasks such as stylized processing, local illumination enhancement and image sharpening, can be accomplished within the unified variational framework. Experimental results verify the high flexibility and efficiency of the proposed model.

Image editing by object-aware optimal boundary searching and mixed-domain composition

When combining very different images which often contain complex objects and backgrounds,producing consistent compositions is a challenging problem requiring seamless image editing. In this paper, we propose a general approach, called objectaware image editing, to obtain consistency in structure,color, and texture in a unified way. Our approach improves upon previous gradient-domain composition in three ways. Firstly, we introduce an iterative optimization algorithm to minimize mismatches on the boundaries when the target region contains multiple objects of interest. Secondly, we propose a mixeddomain consistency metric for measuring gradients and colors, and formulate composition as a unified minimization problem that can be solved with a sparse linear system. In particular, we encode texture consistency using a patch-based approach without searching and matching. Thirdly, we adopt an objectaware approach to separately manipulate the guidance gradient fields for objects of interest and backgrounds of interest, which facilitates a variety of seamless image editing applications. Our unified method outperforms previous state-of-the-art methods in preserving global texture consistency in addition to local structure continuity.

Instant Edit Propagation on Images Based on Bilateral Grid

The ability to quickly and intuitively edit digital content has become increasingly important in our everyday life.However,existing edit propagation methods for editing digital images are typically based on optimization with high computational cost for large inputs.Moreover,existing edit propagation methods are generally inefficient and highly time-consuming.Accordingly,to improve edit efficiency,this paper proposes a novel edit propagation method using a bilateral grid,which can achieve instant propagation of sparse image edits.Firstly,given an input image with user interactions,we resample each of its pixels into a regularly sampled bilateral grid,which facilitates efficient mapping from an image to the bilateral space.As a result,all pixels with the same feature information(color,coordinates)are clustered to the same grid,which can achieve the goal of reducing both the amount of image data processing and the cost of calculation.We then reformulate the propagation as a function of the interpolation problem in bilateral space,which is solved very efficiently using radial basis functions.Experimental results show that our method improves the efficiency of color editing,making it faster than existing edit approaches,and results in excellent edited images with high quality.

Controllable image generation based on causal representation learning

Artificial intelligence generated content(AIGC)has emerged as an indispensable tool for producing large-scale content in various forms,such as images,thanks to the significant role that AI plays in imitation and production.However,interpretability and controllability remain challenges.Existing AI methods often face challenges in producing images that are both flexible and controllable while considering causal relationships within the images.To address this issue,we have developed a novel method for causal controllable image generation(CCIG)that combines causal representation learning with bi-directional generative adversarial networks(GANs).This approach enables humans to control image attributes while considering the rationality and interpretability of the generated images and also allows for the generation of counterfactual images.The key of our approach,CCIG,lies in the use of a causal structure learning module to learn the causal relationships between image attributes and joint optimization with the encoder,generator,and joint discriminator in the image generation module.By doing so,we can learn causal representations in image’s latent space and use causal intervention operations to control image generation.We conduct extensive experiments on a real-world dataset,CelebA.The experimental results illustrate the effectiveness of CCIG.

A Survey of Image Synthesis and Editing with Generative Adversarial Networks

This paper presents a survey of image synthesis and editing with Generative Adversarial Networks（GANs）. GANs consist of two deep networks, a generator and a discriminator, which are trained in a competitive way. Due to the power of deep networks and the competitive training manner, GANs are capable of producing reasonable and realistic images, and have shown great capability in many image synthesis and editing applications.This paper surveys recent GAN papers regarding topics including, but not limited to, texture synthesis, image inpainting, image-to-image translation, and image editing.

Free Appearance-Editing with Improved Poisson Image Cloning

In this paper, we present a new edit tool for the user to conveniently preserve or freely edit the object appearance during seamless image composition. We observe that though Poisson image editing is effective for seamless image composition. Its color bleeding （the color of the target image is propagated into the source image） is not always desired in applications, and it provides no way to allow the user to edit the appearance of the source image. To make it more flexible and practical, we introduce new energy terms to control the appearance change, and integrate them into the Poisson image editing framework. The new energy function could still be realized using efficient sparse linear solvers, and the user can interactively refine the constraints. With the new tool, the user can enjoy not only seamless image composition, but also the flexibility to preserve or manipulate the appearance of the source image at the same time. This provides more potential for creating new images. Experimental results demonstrate the effectiveness of our new edit tool, with similar time cost to the original Poisson image editing.

Facial Image Attributes Transformation via Conditional Recycle Generative Adversarial Networks

This study introduces a novel conditional recycle generative adversarial network for facial attribute transfor- mation, which can transform high-level semantic face attributes without changing the identity. In our approach, we input a source facial image to the conditional generator with target attribute condition to generate a face with the target attribute. Then we recycle the generated face back to the same conditional generator with source attribute condition. A face which should be similar to that of the source face in personal identity and facial attributes is generated. Hence, we introduce a recycle reconstruction loss to enforce the final generated facial image and the source facial image to be identical. Evaluations on the CelebA dataset demonstrate the effectiveness of our approach. Qualitative results show that our approach can learn and generate high-quality identity-preserving facial images with specified attributes.

Image recoloring using geodesic distance based color harmonization

In this paper, we present a computationally simple yet effective image recoloring method based on color harmonization. Our method permits the user to obtain recolored results interactively by rotating a harmonious template after completing color harmonization. Two main improvements are made in this paper. Firstly, we give a new strategy for finding the most harmonious scheme, in terms of finding the template which best matches the hue distribution of the input image. Secondly, in order to achieve spatially coherent harmonization, geodesic distances are used to move hues lying outside the harmonious sectors to inside them. Experiments show that our approach can produce higher-quality visually pleasing recolored images than existing methods. Moreover, our method is simple and easy to implement, and has good runtime performance.

Feature-preserving color pencil drawings from photographs

Color pencil drawing is well-loved due to its rich expressiveness.This paper proposes an approach for generating feature-preserving color pencil drawings from photographs.To mimic the tonal style of color pencil drawings,which are much lighter and have relatively lower saturation than photographs,we devise a lightness enhancement mapping and a saturation reduction mapping.The lightness mapping is a monotonically decreasing derivative function,which not only increases lightness but also preserves input photograph features.Color saturation is usually related to lightness,so we suppress the saturation dependent on lightness to yield a harmonious tone.Finally,two extremum operators are provided to generate a foreground-aware outline map in which the colors of the generated contours and the foreground object are consistent.Comprehensive experiments show that color pencil drawings generated by our method surpass existing methods in tone capture and feature preservation.

Lighting transfer across multiple views through local color transforms

We present a method for transferring lighting between photographs of a static scene. Our method takes as input a photo collection depicting a scene with varying viewpoints and lighting conditions.We cast lighting transfer as an edit propagation problem, where the transfer of local illumination across images is guided by sparse correspondences obtained through multi-view stereo. Instead of directly propagating color, we learn local color transforms from corresponding patches in pairs of images and propagate these transforms in an edge-aware manner to regions with no correspondences. Our color transforms model the large variability of appearance changes in local regions of the scene, and are robust to missing or inaccurate correspondences. The method is fully automatic and can transfer strong shadows between images. We show applications of our image relighting method for enhancing photographs, browsing photo collections with harmonized lighting, and generating synthetic time-lapse sequences.

Reference-guided structure-aware deep sketch colorization for cartoons

Digital cartoon production requires extensive manual labor to colorize sketches with visually pleasant color composition and color shading.During colorization,the artist usually takes an existing cartoon image as color guidance,particularly when colorizing related characters or an animation sequence.Reference-guided colorization is more intuitive than colorization with other hints,such as color points or scribbles,or text-based hints.Unfortunately,reference-guided colorization is challenging since the style of the colorized image should match the style of the reference image in terms of both global color composition and local color shading.In this paper,we propose a novel learning-based framework which colorizes a sketch based on a color style feature extracted from a reference color image.Our framework contains a color style extractor to extract the color feature from a color image,a colorization network to generate multi-scale output images by combining a sketch and a color feature,and a multi-scale discriminator to improve the reality of the output image.Extensive qualitative and quantitative evaluations show that our method outperforms existing methods,providing both superior visual quality and style reference consistency in the task of reference-based colorization.