基于布局图的多物体场景新视角图像生成网络

Multi-object scenes novel view synthesis via layout projection

新视角图像生成任务指通过多幅参考图像,生成场景新视角图像。然而多物体场景存在物体间遮挡,物体信息获取不全,导致生成的新视角场景图像存在伪影、错位问题。为解决该问题,提出一种借助场景布局图指导的新视角图像生成网络,并标注了全新的多物体场景数据集(multi-objects novel view synthesis,MONVS)。首先,将场景的多个布局图信息和对应的相机位姿信息输入到布局图预测模块,计算出新视角下的场景布局图信息;然后,利用场景中标注的物体边界框信息构建不同物体的对象集合,借助像素预测模块生成新视角场景下的各个物体信息;最后,将得到的新视角布局图和各个物体信息输入到场景生成器中构建新视角下的场景图像。在MONVS和ShapeNet cars数据集上与最新的几种方法进行了比较,实验数据和可视化结果表明,在多物体场景的新视角图像生成中,所提方法在两个数据集上都有较好的效果表现,有效地解决了生成图像中存在伪影和多物体在场景中位置信息不准确的问题。

The task of novel view synthesis refers to generating a new perspective image of the scene through multiple reference images.However,there are occlusions between objects in multi-object scenes,and object information cannot be fully obtained,resulting in artifacts and dislocation problems in the generated new-view scene images.In order to solve this problem,this paper proposed a new perspective image generation network guided by the scene layout map,and annotated a new MONVS.Firstly,it input multiple layout information of the scene and the corresponding camera pose information into the layout pre-diction module,and calculated the layout information of the scene under a new perspective.Then,it used the bounding box information of the objects marked in the scene to construct an object set of different objects,and used the pixel prediction module to generate the information of each object in the new perspective scene.Finally,it input the obtained new perspective layout and various object information into the scene generator to construct a scene image under the new perspective.Compared with the latest methods on the MONVS and ShapeNet cars data sets,experimental data and visualization results show that in the new perspective image generation of multi-object scenes,the proposed method has good performance on both data sets.It effectively solved the problem of artifacts in the generated image and inaccurate position information of multiple objects in the scene.