基于生成对抗网的中国山水画双向解码特征融合外推算法

GAN-Based Bidirectional Decoding Feature Fusion Extrapolation Algorithm of Chinese Landscape Painting

研究基于生成对抗网的中国山水画的边界外推问题.现有的图像外推方法主要是针对草地、天空等内容比较单一、纹理比较规范的自然场景进行的,直接将其应用于内容较为复杂、层次丰富、笔触变化多样的中国山水画外推会出现外推内容模糊、与原有图像边界语义不一致等现象.针对上述问题,基于生成对抗网的思想,提出一种新的生成对抗网的双向解码特征融合网络(bidirectional decoding feature fusion generative adversarial network,BDFF-GAN).网络在生成器设计方面,以现有的U型网络(U-Net)为基础,增加一个多尺度解码器,构建一种双向解码特征融合的生成器UY-Net.多尺度解码器抽取编码器不同层级的特征进行交叉互补的组合,增强了不同尺度特征之间的连接交融;同时每一层双向解码的结果还通过条件跳跃连接进一步相互融合.UY-Net设计上的这2个特点有利于网络对山水画不同粒度的语义特征和笔触形态的传递与学习.在鉴别器设计方面,采用全局鉴别器和局部鉴别器相结合的架构,全局鉴别器将整幅山水画作为输入来控制外推结果的全局一致性,局部鉴别器将原有山水画与外推山水画交界处周围的小区域作为输入以提高外推部分与原画作的连贯性和细节生成质量.实验结果表明,与其他方法相比较,所提算法较好地学习到了山水画的语义特征和纹理信息,外推结果在语义内容的连贯性和笔触纹理结构的自然性方面都有更好的表现.此外,还设计了一种新的用户交互方式,该方式通过外推边界引导线的形式控制外推部分的轮廓走向,从而实现了布局可调的山水画外推效果,扩展了上述BDFF-GAN网络的生成多样性和应用互动性.

Some extrapolation methods for Chinese landscape painting based on generative adversarial network is proposed in this paper.Existing image extrapolation methods are mainly designed for natural images with large-scale regions containing same objective in each one and with standardized textures,such as grass and sky.They often suffer from blur and boundary semantic inconsistency in extrapolated regions when they are applied to Chinese landscape painting that have complex details,rich gradations and various strokes.To address those problems,a new bidirectional decoding feature fusion network based on generative adversarial network(BDFF-GAN)is proposed.The generator,named UY-Net,is designed with the architecture of U-Net and a multi-scale decoder,which can achieve the function of bidirectional decoding features fusion.Features from different layers of the encoder are assigned to corresponding layers of the multi-scale decoder,where the first-stage feature fusion is achieved by concatenation operations and therefore the connections between features of different scales are enhanced.On the other hand,decoded features from U-Net part and the multi-scale decoder part at same scales are fused by skipping connections to further improve the performance of the generator.Benefiting from the subtle architecture,UY-Net can perform well at semantic features and stroke transmission as well as learning.Moreover,multi-discriminator strategy is adopted in our method.A global discriminator takes the whole result image as the input to control the global consistency,and a local discriminator takes the patch from the junction of source image part and extrapolated part as the input to improve the coherence and details.Experimental results show that BDFF-GAN performs well at semantic features and textures learning with regards to landscape paintings and outperforms existing methods in terms of the semantic content coherence and the naturalness of texture structure with regards to strokes.In addition,we provide an interface that allows users to control the outline of the extrapolated part by boundary guide lines,which achieves the controllability for the layout of extrapolated part and expands the generation diversity and application interactivity of BDFF-GAN.