附加偏见预测器辅助的均衡化场景图生成

Balanced scene graph generation assisted by an additional biased predictor

场景图是以场景中的物体为结点、以物体之间的关系为边构成的图结构,在视觉与语言交互理解和推理相关任务中具有广泛的应用前景.近年来,场景图自动生成逐渐受到关注,但生成结果中对于关系的描述受到长尾分布带来的偏见的影响,偏向于样本量较大的头部关系.然而头部关系往往过于空泛,描述不够准确,容易造成误解.由于这种关系价值不高,生成的场景图近似于退化为场景中物体信息的堆叠,不利于其他应用在图结构上进行结构化推理.为了使场景图生成器在这种不均衡的数据条件下,能够更均衡地学习,给出更加多样化的特别是尾部的更准确的关系,本文提出一种附加偏见预测器(additional biased predictor,ABP)辅助的均衡化学习方法.该方法利用一条有偏见的关系预测分支,令场景图生成器抑制自身对头部关系的偏好,并更加注重尾部关系的学习.场景图生成器需要为指定的一对物体预测关系,这是一种实例级的关系预测,与之相比,有偏分支以更简洁的方式预测出图像中的关系信息,即不指定任何一对物体,直接预测出图像中存在的关系,这是一种区域级的关系预测.为此,本文利用已有的实例级的关系标注,设计算法自动构造区域级的关系标注,以此来训练该有偏分支,使其具有区域级关系预测的能力.在不同场景图生成器上应用ABP方法,并在多个公开数据集(Visual Genome,VRD和OpenImages等)上进行实验,结果表明,ABP方法具有通用性,应用ABP方法训练得到的场景图生成器能够预测出更加多样化的、更准确的关系,进而生成更有价值、更实用的场景图.

A scene graph is a structural representation of a scene comprising the objects as nodes and relationships between any two objects as edges. The scene graph is widely adopted in high-level vision language and reasoning applications. Therefore, scene graph generation has been a popular topic in recent years. However, it is limited by bias due to the long-tailed distribution among the relationships. Scene graph generators prefer to predict the head predicates, which are ambiguous and less precise. It makes the scene graph convey less information and degenerate into the stacking of objects, restricting other applications from reasoning on the graph.To make the generator predict more diverse relationships and provide a precise scene graph, we propose an additional biased predictor(ABP)-assisted balanced learning method. This method introduces an extra relationship prediction branch that is especially affected by the bias to make the generator pay more attention to the tail predicates rather than the head ones. Compared to the scene graph generator that predicts relationships between object pairs, the biased branch predicts the relationships without being assigned a certain object pair of interest,which is more concise. To train this biased branch, the region-level relationship annotation is constructed using the instance-level relationship annotation automatically. Extensive experiments on popular datasets, i.e., Visual Genome, VRD, and OpenImages, show that the ABP is effective on different scene graph generators. Besides, it makes the generator predict more diverse and accurate relationships and provides a more balanced and practical scene graph.