基于自适应正则化的无偏场景图生成方法

Unbiased Scene Graph Generation Based on Adaptive Regularization Algorithm

场景图生成旨在给定一张图片,通过目标检测模块得到实体和实体间关系的视觉三元组形式,即主语、关系和宾语,构建语义结构化表示。场景图可应用于图像检索和视觉问答等下游任务。然而,由于数据集中的实体间关系呈长尾分布,因此现有模型在预测关系时更偏向于粗粒度的头部关系。这样的场景图无法对下游任务起到辅助性作用。以往工作普遍采用再平衡策略,如重采样和重加权的方法,来解决长尾问题。但模型反复学习尾部关系样本,易出现过拟合现象。为了解决上述问题,文中提出了一种自适应正则化无偏场景图生成方法。具体来说,该方法通过设计一个基于先验关系频率的正则项,自适应地调整模型全连接分类器权重,从而实现对模型的平衡预测。所提方法在场景图VG(Visual Genome)数据集上进行了实验,实验结果表明,该方法不仅能防止模型过拟合,也能缓解关系长尾分布问题对场景图生成的负面影响,且最先进的场景图生成方法在结合所提方法后能更有效地改善无偏场景图生成的性能。

The purpose of scene graph generation is to give a picture,obtain the visual triplet form of entities and relationships between entities through the object detection module,namely subject,relationship and object,and construct a semantic structured representation.Scene graphs can be applied to downstream tasks such as image retrieval and visual question answering.However,due to the longtail distribution of relationships between entities in the dataset,existing models tend to predict coarse grained head relationships.Such scene graph cannot play an auxiliary role for downstream tasks.Previous works generally adopt rebalancing strategies such as resampling and reweighting to solve the long tail problem.However,because the models repeatedly learn the tail relationship samples,it is prone to overfitting.In order to solve the above problems,an adaptive regularized unbiased scene graph generation method is proposed in this paper.Specifically,the method adaptively adjusts the weights of full connected classifier of the model by designing a regularization term based on the prior relation frequency,so as to achieve the prediction of model balance.The proposedmethod is tested on Visual Genome dataset,and the experimental results show that it can not only prevent the model from overfitting,but also alleviate the negative impact of the longtail distribution problem on the scene graph generation,and the state-of-the-artscene graph generation methods combined with the proposed method can more effectively improve the performance of unbiased scene graph generation.