改进深度卷积生成式对抗网络的文本生成图像

Text-to-image synthesis based on modified deep convolutional generative adversarial network

针对深度卷积生成式对抗网络(DCGAN)模型高维文本输入表示的稀疏性导致以文本为条件生成的图像结构缺失和图像不真实的问题,提出了一种改进深度卷积生成式对抗网络模型CA-DCGAN。采用深度卷积网络和循环文本编码器对输入的文本进行编码,得到文本的特征向量表示。引入条件增强(CA)模型,通过文本特征向量的均值和协方差矩阵产生附加的条件变量,代替原来的高维文本特征向量。将条件变量与随机噪声结合作为生成器的输入,并在生成器的损失中额外加入KL损失正则化项,避免模型训练过拟合,使模型可以更好的收敛,在判别器中使用谱约束(SN)层,防止其梯度下降太快造成生成器与判别器不平衡训练而发生模式崩溃的问题。实验验证结果表明:所提模型在Oxford-102-flowers和CUB-200数据集上生成的图像质量较alignDRAW、GAN-CLS、GAN-INT-CLS、StackGAN(64×64)、StackGAN-v1(64×64)模型更好且接近于真实样本,初始得分值最低分别提高了10.9%和5.6%,最高分别提高了41.4%和37.5%,FID值最低分别降低了11.4%和8.4%,最高分别降低了43.9%和42.5%,进一步表明了所提模型的有效性。

When high-dimensional texts are adopted as input,images generated by the previously proposed deep convolutional generative adversarial network(DCGAN)model usually suffer from distortions and structure degradation due to the sparsity of texts,which seriously poses a negative impact on the generative performance.To address this issue,an improved deep convolutional generative adversarial network model,CA-DCGAN is proposed.Technically,a deep convolutional network and a recurrent text encoder are simultaneously employed to encode the input text so that the corresponding text embedding representation can be obtained.Then,a conditional augmentation(CA)model is introduced to generate an additional condition variable to replace the original high-dimensional text feature.Finally,the conditional variable and random noise are combined as the input of the generator.Meanwhile,to avoid over-fitting and promote the convergence,we introduce a KL regularization term into the generator’s loss.Moreover,we adopt a spectral normalization(SN)layer in the discriminator to prevent the mode collapse caused by the unbalanced training due to the fast gradient descent of the discriminator.The experimental verification results show that the proposed model on the Oxford-102-flowers and CUB-200 datasets is better than that of alignDRAW,GAN-CLS,GAN-INT-CLS,StackGAN(64×64),StackGAN-vl(64×64)in terms of the quality of generated images.The results show that the lowest inception score increased by 10.9%and 5.6%respectively,the highest inception score increased by 41.4%and 37.5%respectively,while the lowest FID index value decreased by 11.4%and 8.4%respectively,the highest FID index value decreased by 43.9%and 42.5%respectively,which further validate the effectiveness of the proposed method.