基于词序列拼积木模型的图像句子标注研究

Image Annotation by Sentences Based on Word Sequence Blocks Building Model

用句子标注图像,建立图像与文本间的跨媒体关联,以提升信息检索准确率,改善用户检索交互体验.利用KDES模型抽取图像特征,在多核学习模型中融合出MK-KDES特征,准确刻画图像视觉特性;设计自然语言生成模型:词序列拼积木,评估单词与图像内容的相关性,优选单词,并根据单词间的语义相关性与句法模式约束,将单词组合成N元词序列;把N元词序列输入模板生成句子.结果表明:MK-KDES-1特征聚焦于图像的纹理及形状视觉特性,它是改善句子BLEU-1评分的关键;而单词间的语义相关性与句法模式约束是提升句子BLEU-2评分的重要前提.

Based on image annotation by sentences,the cross-media correlations between the images and the texts were constructed to improve the information retrieval accuracy and users＇retrieval experiences ultimately.The KDES model was applied to extract image features effectively and the MK-KDES features were obtained in turn by fusing the extracted features in the multiple kernel learning model to interpret the key visual characteristics of the images.A new natural language generation model named word sequence blocks building（WSBB）was designed to evaluate the semantic correlations between the words and the images.And several key words were summarized for generating sentences.According to the semantic correlations and syntactic mode constraints between words,many N gram word sequences were made up of those summarized words to better interpret the images.Finally,sentences were generated by inputting the N gram word sequences into the templates.Experimental results show that,the MK-KDES-1 feature can focus on describing the key texture and shape characteristics of the images,which helps to improve the BLEU-1 scores.Moreover,semantic correlation between the words is an important premise of improving the BLEU-2 scores as well as the syntactic mode constraints.