图像语义相似性网络的文本描述方法

IMAGE CAPTION BASED ON IMAGE SEMANTIC SIMILARITY NETWORK

图像的文本化描述解决图像理解的高级语义问题。由于存在语义鸿沟,导致自动生成的文本与图像内容有较大差异,浅层神经网络构建的语言模型很难生成通顺的语句。为此,提出图像语义相似性神经网络,在递归神经网络的输出层之后添加全连接网络,引入图像间的视觉相似性和文本相似性信息,从而在预测图像的文本描述时保持相似图像的有效语义信息。增加栈式隐层和普通隐层的深度来提高语言模型的学习能力,最终得到接近自然语言的文本。实验结果表明,该方法在BLEU、ROUGE、METEOR和CIDEr等评价指标上均取得较好的效果,能够生成符合图像内容的高质量文本描述。

Image caption solves the problem of high level semantic image understanding. Due to the semantic gap, there are differences between the generative captions and images, and the shallow neural network based language model is hard to generate smooth sentences. To solve this problem, this paper proposes image semantic similarity neural network, which adds full connected layers after the output layer of recurrent neural network, and thus introduce the visual similarity and semantic similarity information between images. Besides, this method increases the depth of stacked hidden layers and common hidden layers to improve the learning ability of language model, and finally obtain image captions close to human natural language. Experiments show that the proposed method get great respond from BLEU, ROUGE, METEOR and CIDEr assessment criteria by obtaining high quality captions.