融合多时间维度视觉与语义信息的图像描述方法

Image Captioning Method for Fusing Multi-temporal Dimensional Visual and Semantic Information

传统的图像描述方法仅使用当前时刻的视觉信息和语义信息来生成预测词,而没有考虑过去时刻的视觉信息和语义信息,从而导致模型输出的信息在时间维度上比较单一,因此生成的描述语句在准确性上有所欠缺。针对此问题,提出一种融合多时间维度视觉与语义信息的图像描述方法,有效地融合了过去时刻的视觉信息和语义信息,并设计一种门控机制动态地对两种信息进行选择利用。在MSCOCO数据集上进行实验验证,结果表明该方法能够更准确地生成描述语句,和当前最主流的图像描述方法进行对比,性能在各项评价指标上都得到了可观的提升。

Traditional image captioning methods use only the visual and semantic information of the current moment to generate prediction words without considering the visual and semantic information of the past moments,which leads to the output of the model to be relatively homogeneous in terms of temporal dimension.As a result,the generated captioning is lacking in terms of accuracy.To address this problem,an image captioning method that fuses multi-temporal dimensional visual and semantic information is proposed,which effectively fuses visual and semantic information of past moments and designs a gating mechanism to dynamically select both kinds of information.Experimental validation on the MSCOCO dataset shows that the method is able to generate captioning more accurately,and the performance is considerably improved in all evaluation metrics when compared with the most current state-of-the-art image captioning methods.