基于注意力和视觉语义推理的枸杞虫害检索

Lycium Barbarum Pest Retrieval Based on Attention and Visual Semantic Reasoning

针对传统作物虫害检索模态单一的问题,将注意力与视觉语义推理相结合,对常见的17种枸杞虫害进行图文跨模态检索研究。首先利用Faster R-CNN+ResNet101实现注意力机制来提取枸杞虫害图像局部细粒度信息;接着,引入视觉语义推理,建立图像区域连接并采用图卷积网络(GCN)进行区域关系推理来增强区域表示;然后,进一步进行全局语义推理,选择具有判别性的特征,过滤掉不重要的内容,以捕获更多的关键语义信息;最后通过模态交互深入挖掘枸杞虫害图像和文本不同模态间的语义关联。在自建的枸杞虫害数据集上,采用平均准确率均值(MAP)作为评价指标对所提方法进行对比实验和消融实验。实验结果表明,图检文和文检图的平均MAP值达到了0.522,与8种主流方法相比提升了0.048~0.244,具有更好的检索效果。

Aiming at the problem that traditional retrieval model on pest has a single mode,this paper uses a cross-modal retrieval method for 17 kinds of common lycium pests in image and text modal,which integrates attention mechanism and visual semantic reasoning.First,use Faster R-CNN+ResNet101 to realize the attention mechanism to extract local fine-grained information of wolfberry pest images.Then further introduce vision semantic reasoning to build the image region connections and use convolutional network GCN for region relation reasoning to enhance area representation.In addition,global semantic reasoning is performed by enhancing semantic correlation between regions,selecting discriminant features and filtering out unimportant information to capture more key semantic information.Finally,the semantic association between different modalities of lycium barbarum pest image and text is deeply explored through modal interaction.On the self-built lycium barbarum pest dataset,the average accuracy(MAP)is used as the evaluation index to carry out comparative experiment and ablation experiment.Experimental results demonstrate that the averaged MAPof the proposed method in the self-built lycium pest dataset achieves 0.522,compared with the eight mainstream methods,the average MAP of the method improves by 0.048 to 0.244,and it has better retrieval effect.