融合PointNet和3D-LMNet的单幅图像三维重建及语义分割

3D Reconstruction and Semantic Segmentation Method Combining PointNet and 3D-LMNet from Single Image

由单幅图像重建三维结构并感知三维对象的语义理解极具挑战性。针对单幅图像难以直接生成三维重建点云问题,提出一种融合PointNet与3D-LMNet的联合优化网络模型进行三维重建并完成语义分割。基于3D-LMNet网络进行训练生成三维点云,并完成局部分割,同时,对网络损失函数进行联合优化来预测分割点云。通过分割点云的语义信息改善重建效果,生成带有语义分割信息的三维点云重建模型。针对联合训练中真值点云和预测点云类别标签无点对点的对应关系问题,引入联合优化损失函数来提高重建和分割效果,生成最终三维重建模型。通过在ShapeNet数据集上实验验证,并与PointNet和3D-LMNet单独训练相比,所提模型在平均交并比(mIoU)上提高了4.23%,在倒角距离(CD)和EMD(earth mover’s distance)上分别降低了7.97%和6.04%,联合优化网络明显改善了重建和分割的点云模型。

It is very challenging to reconstruct the 3D structure from a single image and perceive the semantic information of 3D objects.Aiming at the problem that it is difficult to directly generate a 3D reconstruction model from a single image input,a joint optimization network model combining PointNet and 3DLMNet is proposed for single image 3D reconstruction and semantic segmentation.First,a 3D point cloud is generated by training based on the 3DLMNet network,and then local segmentation is performed.Meanwhile,the network loss function is jointly optimized to predict the segmented 3D point cloud.Then,the reconstruction effect is improved through the semantics information of segmented point cloud,and a 3D point cloud reconstruction model is generated with semantic segmentation information.Finally,in view of the problem that there is no pointtopoint correspondence between the true value point cloud and the predicted point cloud category label during the joint training,the joint optimization loss function is introduced into the joint optimization network to improve the reconstruction and segmentation effect,and the 3D reconstructed model is made.Through verification on the ShapeNet dataset,and comparation with PointNet and 3DLMNet training,the model in this paper improves mean intersection over union(mIoU)by 4.23%,and reduces chamfer distance(CD)and earth mover’s distance(EMD)by 7.97%and 6.04%,respectively.The joint optimization network significantly improves the reconstruction and segmented point cloud model.