基于物体检测及关系推理的视觉问答方法研究

Research on Visual Question and Answering Method Based on Object Detection and Relational Reasoning

大多数图像描述及视觉问答任务中,主要工作是对图像数据的拟合分类,而缺乏图像中物体之间的关系推理,导致描述图像或回答问题时准确率不高.为解决该问题,本文提出一种基于物体检测及关系推理的视觉问答模型.该模型由上游网络和下游网络两部分组成,上游网络采用极快速区域卷积神经网络,下游网络由多层感知机组成的多模态特征融合推理网络构成.上游网络对图像进行物体检测与特征提取,利用长短期记忆网络对提问的问题进行信息提取以嵌入下游网络;下游网络对问题和图像的特征进行融合和推理,进而得出答案.采用数据集CLEVR进行模型训练与视觉问答测试,实验结果表明,本模型与其他已有模型相比,图像中物体之间关系推理的准确率获得了提高,达到98.96%.

In most image description and visual question answering tasks,the main task is to fit and classify the image data.The lack of reasoning relationship between objects in the image leads to low accuracy in describing images or answering questions.Visual question answering model based on object detection and relational reasoning is proposed to overcome this defect.The model is composed of upstream network and downstream network.The upstream network adopts Faster Regional Convolutional Neural Network,and the downstream network consists of Multimodality Feature Fusion Inference Network,and the Network is composed of Multilayer Perceptron.First,image target detection and feature extraction are practiced in the upstream network,and problem information is extracted in Long Short-Time Memory network to embed in the downstream network.Then problem text features and image features are fused and inferred in the downstream network to obtain the answers.Model training and visual question answering test are performed on the dataset CLEVR.The results show that the accuracy of the model is higher than other existing models,the accuracy of inference between objects in the image is improved to 98.96%.