ST-Rec3D:基于结构和目标感知的三维重建

ST-Rec3D:a structure and target-aware 3D reconstruction

基于视图的三维重建旨在从二维图像恢复出其对应的三维形状。现有方法主要通过编码器-解码器结构,结合二元交叉熵函数及其变形,完成三维重建,取得较好的重建结果。然而,编码器在编码过程中缺乏对输入视图的结构感知能力,造成重建的三维模型几何细节不准确;以二元交叉熵函数为主的损失函数在体素分布不均衡的情况下,目标感知能力较差,导致其重建结果存在断裂、缺失等不完整性问题。针对此类问题,提出了一种具有结构和目标感知能力的三维重建网络(ST-Rec3D),以单视图或多视图为输入,由粗到细地重建出三维模型;结合注意力机制提出了一种具有空间结构感知能力的编码器,即结构编码器,以充分捕捉输入视图中的空间结构信息,有效感知重建物体的几何细节;将IoU损失引入到三维体素模型重建中,在体素分布不均衡的情况下,精准感知目标物体,确保重建物体的完整性和准确性。在ShapeNet和Pix3D数据集上的对比结果表明,ST-Rec3D在单视图和多视图上重建的三维模型的完整性和准确性均优于当前方法。

Image-based 3D reconstruction is the process of producing 3D representations of an object based on its single or multiple images.Existing methods for 3D reconstruction can directly learn to transform image features into 3D representations,using encoder-decoder structure,combined with binary cross entropy function and its deformation.However,the encoder cannot extract enough information from images to reconstruct high-quality 3D shapes,resulting in inaccurate Geometric details of reconstructed 3D objects.The loss functions based on the binary cross entropy function underperforms in target perception when the voxel distribution is imbalanced,leading to problems of incompleteness such as fractures and missing in the reconstruction results.To address these problems,a structure and target-aware 3D object reconstruction framework was proposed for single-view and multi-view 3D reconstruction,named ST-Rec3D.Combined with attention mechanism,we designed an encoder with a spatial perception structure,namely structure-aware encoder.In doing so,the spatial structure information could be fully captured in the input image and the local details of the reconstructed object could be effectively perceived.The utilization of IoU loss in the 3D voxel reconstruction,in the case of uneven voxel distribution,could accurately perceive the target object to ensure the integrity and accuracy of the reconstructed object.Experimental results demonstrate that ST-Rec3D can give a significant boost to reconstruction quality and outperform state-of-the-art methods on the ShapeNet and Pix3D.