体现用户意图和风格的图像描述生成

Image Captioning According to User’s Intention and Style

现有的图像描述模型大多不能根据用户的意图和用语风格生成个性化的描述。针对这一问题,本文提出一种能体现用户意图和风格的个性化图像描述方法。首先,构建一个关于场景中目标、目标属性以及目标间关系的结构图,通过该图来控制用户所希望表达的目标对象、目标属性以及各目标之间的相互关系;然后,在编码器中加入多关系图卷积神经网络对场景的上下文信息进行编码,并利用图流动注意力来控制描述的侧重点;最后,在生成语句时加入用户风格控制模块,即利用关键词搜索生成包含性别、年龄、受教育程度等信息的用户画像,并结合该画像来控制风格生成器,提取对应的风格样式,最终生成体现用户意图和风格的个性化图像描述。在MSCOCO和FlickrStyle数据集上的实验结果表明,所提出的方法能较好地生成个性化和多样性图像描述语句。

Most of the image captioning models are individuality-agnostic,which cannot generate an individual description according to the user’s intention and language style.To address the above problem,a personalized image captioning model is established in this paper by using fine-grained scene control graph and the style control factors to represent user’s intention and style of speaking,respectively.Firstly,construct a scene control graph,including the object,object attributes and the relationship objects in the scene,which can control the object,object attributes and the relationship between object.Secondly,a multi-graph convolutional neural network is used to encode the context information of the scene,and graph flow attention is employed to control the focus of the description.Then,add the style control module when generating sentences,that is,use keyword search to generate user profile according to user’s gender,age,education level and other information.Finally,the style generator extracts the corresponding style pattern according to the user profile,and the language decoder outputs a personalized image caption.The experimental results on MSCOCO dataset and FlickrStyle dataset show that the proposed method can generate personalized and diverse image caption sentences.