知识型视觉问答研究综述

Knowledge-based Visual Question Answering:A Survey

视觉问答作为人工智能完备性和视觉图灵测试的重要呈现形式,加上其具有潜在的应用价值,受到了计算机视觉和自然语言处理两个领域的广泛关注。知识在视觉问答中发挥着重要作用,特别是在处理复杂且开放的问题时,推理知识和外部知识对获取正确答案极为关键。蕴含知识的问答机制被称为知识型视觉问答,目前还没有针对知识型视觉问答的系统性调查。面向视觉问答中的知识参与方式和表达形式的研究能够有效填补知识型视觉问答体系中在文献综述方面存在的缺口。文中对知识型视觉问答的各组成单元进行了调查,对知识的存在形态进行了研究,提出了知识层级概念。进一步地,针对视觉特征提取、语言特征提取和多模态融合过程中的知识参与方式和表达形式进行了归纳和总结,并对未来发展趋势及研究方向进行了探讨。

As an important presentation form of the completeness of artificial intelligence and the visual Turing test,visual question answering(VQA),coupled with its potential application value,has received extensive attention from computer vision and na-tural language processing.Knowledge plays an important role in visual question answering,especially when dealing with complex and open questions,reasoning knowledge and external knowledge are critical to obtaining correct answers.The question and answer mechanism that contains knowledge is called knowledge-based visual question answering(Kb-VQA).At present,systematic investigations on Kb-VQA have not been discovered.Research on knowledge participation methods and expression forms in VQA can effectively fill the gaps in the literature review in the knowledge-based visual question answering system.In this paper,the constituent units of Kb-VQA are investigated,the existence of knowledge is studied,and the concept of knowledge hierarchy is proposed.Further,the knowledge participation methods and expression forms in the process of visual feature extraction,language feature extraction and multi-modal fusion are summarized,and future development trends and research directions are discussed.