基于Transformer交叉注意力的文本生成图像技术

Text-to-Image Generation Technology Based on Transformer Cross Attention

近年来,以生成对抗网络为基础的从文本生成图像方法的研究取得了一定的进展。文本生成图像技术的关键在于构建文本信息和视觉信息间的桥梁,促进网络模型生成与对应文本描述一致的逼真图像。目前,主流的方法是通过预训练文本编码器来完成对输入文本描述的编码,但这些方法在文本编码器中未考虑与对应图像的语义对齐问题,独立对输入文本进行编码,忽略了语言空间与图像空间之间的语义鸿沟问题。为解决这一问题,文中设计了一种基于交叉注意力编码器的对抗生成网络(CAE-GAN),该网络通过交叉注意力编码器,将文本信息与视觉信息进行翻译和对齐,以捕捉文本与图像信息之间的跨模态映射关系,从而提升生成图像的逼真度和与输入文本描述的匹配度。实验结果表明,在CUB和coco数据集上,与当前主流的方法DM-GAN模型相比,CAE-GAN模型的IS(Inception Score)分数分别提升了2.53%和1.54%,FID (Fréchet Inception Distance)分数分别降低了15.10%和5.54%,由此可知,CAE-GAN模型生成图像的细节更加完整、质量更高。

In recent years,the research on the methods of text to image based on generative adversarial network(GAN) continues to grow in popularity and have made some progress.The key of text-to-image generation technology is to build a bridge between the text information and the visual information,and promote the model to generate realistic images consistent with the corresponding text description.At present,the mainstream method is to complete the encoding of the descriptions of the input text by pre-training the text encoder,but these methods do not consider the semantic alignment with the corresponding image in the text encoder,and adopt the independent encoding of the input text,ignoring the semantic gap between the language space and the image space.To address the problem,in this paper,a generative adversarial network based on the cross-attention encoder(CAE-GAN) is proposed.The network uses a cross-attention encoder to translate and align text information with visual information,and captures the cross-modal mapping relationship between text and image information,so as to improve the fidelity of the gene-rated images and the matching degree with input text description.The experimental results show that,compared with the DM-GAN model,the inception score(IS) of CAE-GAN model increases by 2.53% and 1.54% on CUB and coco datasets,respectively.The fréchet inception distance score decreases by 15.10% and 5.54%,respectively,indicating that the details and the quality of the images generated by the CAE-GAN model are more perfect.