针对多视图三维重建中存在的内存和时间消耗过大、高分辨率重建完整性差等问题,提出一种基于深度学习的多视图重建网络。网络由特征提取模块、级联的Patchmatch模块和深度图优化模块组成。首先,设计U型的特征提取模块,提取多阶段特征图...针对多视图三维重建中存在的内存和时间消耗过大、高分辨率重建完整性差等问题,提出一种基于深度学习的多视图重建网络。网络由特征提取模块、级联的Patchmatch模块和深度图优化模块组成。首先,设计U型的特征提取模块,提取多阶段特征图,并在每个阶段引入相对位置编码的局部自注意力层,捕捉图像中的局部细节和全局上下文,提升网络特征提取性能。其次,设计深度残差网络,通过密集连接和残差结构对特征进行融合,充分利用彩色图像先验知识来约束深度图,提升深度估计的准确性。在公开数据集DTU(Technical University of Denmark)上进行测试,实验结果表明,三维重建质量到了有效的提升,与PatchmatchNet相比在完整性上提升了6.1%,在整体性上提升了2.5%,与其他的SOTA(State-Of-The-Art)方法相比,在完整性和整体性上都得到了较大提升。展开更多
为解决双目视觉三维重建深度图边缘不连续的问题,提出基于加权最小二乘(weighted least squares,WLS)滤波的深度图优化,经双目标定、畸变矫正、立体校正、立体匹配建立三维深度图,加入WLS滤波,通过调整正则项更改约束条件,对梯度较大的...为解决双目视觉三维重建深度图边缘不连续的问题,提出基于加权最小二乘(weighted least squares,WLS)滤波的深度图优化,经双目标定、畸变矫正、立体校正、立体匹配建立三维深度图,加入WLS滤波,通过调整正则项更改约束条件,对梯度较大的区域减少约束,保留图像边缘,对梯度较小的区域平滑处理,去除噪声,采用峰值信噪比、结构相似性指数、平均绝对误差3个参数评价图像质量。评价结果表明:与半全局匹配算法相比,此算法的峰值信噪比增大1.849 dB,图像失真更少,质量更高;结构相似性指数增大0.4151,与原图结构相似性更强;平均绝对误差减小21.5422,还原度更高。重建的深度图视觉效果更好,改善立体匹配不连续的问题,减小匹配误差,使视差图质量更高。展开更多
The depth information of the scene indicates the distance between the object and the camera,and depth extraction is a key technology in 3D video system.The emergence of Kinect makes the high resolution depth map captu...The depth information of the scene indicates the distance between the object and the camera,and depth extraction is a key technology in 3D video system.The emergence of Kinect makes the high resolution depth map capturing possible.However,the depth map captured by Kinect can not be directly used due to the existing holes and noises,which needs to be repaired.We propose a texture combined inpainting algorithm in this paper.Firstly,the foreground is segmented combined with the color characteristics of the texture image to repair the foreground of the depth map.Secondly,region growing is used to determine the match region of the hole in the depth map,and to accurately position the match region according to the texture information.Then the match region is weighted to fill the hole.Finally,a Gaussian filter is used to remove the noise in the depth map.Experimental results show that the proposed method can effectively repair the holes existing in the original depth map and get an accurate and smooth depth map,which can be used to render a virtual image with good quality.展开更多
This manuscript presents a modified algorithm to extract depth information from stereo-vision acquisitions using a correlation based approaches. The main implementation of the proposed method is in the area of autonom...This manuscript presents a modified algorithm to extract depth information from stereo-vision acquisitions using a correlation based approaches. The main implementation of the proposed method is in the area of autonomous Pick & Place, using a robotic manipulator. Current vision-guided robotics is still based on a priori training and teaching steps, and still suffers from long response time. This study uses a stereo triangulation setup where two Charged Coupled Devices CCDs are arranged to acquire the scene from two different perspectives. The study discusses the details to calculate the depth using a correlation matching routine which programmed using a Square Sum Difference SSD algorithm to search for the corresponding points from the left and the right images. The SSD is further modified using an adjustable Region Of Interest ROI along with a center of gravity based calculations. Furthermore, the two perspective images are rectified to reduce the required processing time. The reported error in depth using the modified SSD method is found to be around 1.2 mm.展开更多
针对使用深度神经网络进行多视角图像三维重建时存在特征图对光照变化敏感以及重建不完整的问题,提出了一种融合梯度和高斯过程回归的多视图重建方法.首先,针对光照变化影响提取特征的问题,设计一个融合梯度的特征提取网络.通过对图像...针对使用深度神经网络进行多视角图像三维重建时存在特征图对光照变化敏感以及重建不完整的问题,提出了一种融合梯度和高斯过程回归的多视图重建方法.首先,针对光照变化影响提取特征的问题,设计一个融合梯度的特征提取网络.通过对图像进行独立的梯度计算并在梯度与原图像的基础上使用卷积神经网络提取特征,提高了梯度信息在特征图中的彩响力,增强了特征图对光照变化因素影响的抑制力.其次,针对多视图重建中特征提取步骤只关注当前视图而没有考虑视图间的潜在空间关系的问题,提出一个融合高斯过程回归算法的视图特征增强模块,有效地增益了视图间相关信息对多视立体视觉重建任务的影响,提高了多视立体视觉重建结果的完整度.最后,通过衡量参考图像与相邻图像特征体之间的匹配程度计算不同视图对Costvolume的贡献度,重新构建符合视觉感知的CostVolume.在DTU和Tanks and Temples数据集上进行实验,结果表明,与主流的多视立体视觉重建方法相比,该方法在三维重建的完整度方面有较大提升,并且拥有良好的泛化性.展开更多
文摘针对多视图三维重建中存在的内存和时间消耗过大、高分辨率重建完整性差等问题,提出一种基于深度学习的多视图重建网络。网络由特征提取模块、级联的Patchmatch模块和深度图优化模块组成。首先,设计U型的特征提取模块,提取多阶段特征图,并在每个阶段引入相对位置编码的局部自注意力层,捕捉图像中的局部细节和全局上下文,提升网络特征提取性能。其次,设计深度残差网络,通过密集连接和残差结构对特征进行融合,充分利用彩色图像先验知识来约束深度图,提升深度估计的准确性。在公开数据集DTU(Technical University of Denmark)上进行测试,实验结果表明,三维重建质量到了有效的提升,与PatchmatchNet相比在完整性上提升了6.1%,在整体性上提升了2.5%,与其他的SOTA(State-Of-The-Art)方法相比,在完整性和整体性上都得到了较大提升。
文摘为解决双目视觉三维重建深度图边缘不连续的问题,提出基于加权最小二乘(weighted least squares,WLS)滤波的深度图优化,经双目标定、畸变矫正、立体校正、立体匹配建立三维深度图,加入WLS滤波,通过调整正则项更改约束条件,对梯度较大的区域减少约束,保留图像边缘,对梯度较小的区域平滑处理,去除噪声,采用峰值信噪比、结构相似性指数、平均绝对误差3个参数评价图像质量。评价结果表明:与半全局匹配算法相比,此算法的峰值信噪比增大1.849 dB,图像失真更少,质量更高;结构相似性指数增大0.4151,与原图结构相似性更强;平均绝对误差减小21.5422,还原度更高。重建的深度图视觉效果更好,改善立体匹配不连续的问题,减小匹配误差,使视差图质量更高。
基金Supported by the Key Project of National Natural Science Foundation of China(Nos.60832003 and 61172096)major Project of Shanghai Science and Technology Committee(No.10510500500)the Major Innovation Project of Shanghai Municipal Education Commission
文摘The depth information of the scene indicates the distance between the object and the camera,and depth extraction is a key technology in 3D video system.The emergence of Kinect makes the high resolution depth map capturing possible.However,the depth map captured by Kinect can not be directly used due to the existing holes and noises,which needs to be repaired.We propose a texture combined inpainting algorithm in this paper.Firstly,the foreground is segmented combined with the color characteristics of the texture image to repair the foreground of the depth map.Secondly,region growing is used to determine the match region of the hole in the depth map,and to accurately position the match region according to the texture information.Then the match region is weighted to fill the hole.Finally,a Gaussian filter is used to remove the noise in the depth map.Experimental results show that the proposed method can effectively repair the holes existing in the original depth map and get an accurate and smooth depth map,which can be used to render a virtual image with good quality.
文摘This manuscript presents a modified algorithm to extract depth information from stereo-vision acquisitions using a correlation based approaches. The main implementation of the proposed method is in the area of autonomous Pick & Place, using a robotic manipulator. Current vision-guided robotics is still based on a priori training and teaching steps, and still suffers from long response time. This study uses a stereo triangulation setup where two Charged Coupled Devices CCDs are arranged to acquire the scene from two different perspectives. The study discusses the details to calculate the depth using a correlation matching routine which programmed using a Square Sum Difference SSD algorithm to search for the corresponding points from the left and the right images. The SSD is further modified using an adjustable Region Of Interest ROI along with a center of gravity based calculations. Furthermore, the two perspective images are rectified to reduce the required processing time. The reported error in depth using the modified SSD method is found to be around 1.2 mm.
文摘针对使用深度神经网络进行多视角图像三维重建时存在特征图对光照变化敏感以及重建不完整的问题,提出了一种融合梯度和高斯过程回归的多视图重建方法.首先,针对光照变化影响提取特征的问题,设计一个融合梯度的特征提取网络.通过对图像进行独立的梯度计算并在梯度与原图像的基础上使用卷积神经网络提取特征,提高了梯度信息在特征图中的彩响力,增强了特征图对光照变化因素影响的抑制力.其次,针对多视图重建中特征提取步骤只关注当前视图而没有考虑视图间的潜在空间关系的问题,提出一个融合高斯过程回归算法的视图特征增强模块,有效地增益了视图间相关信息对多视立体视觉重建任务的影响,提高了多视立体视觉重建结果的完整度.最后,通过衡量参考图像与相邻图像特征体之间的匹配程度计算不同视图对Costvolume的贡献度,重新构建符合视觉感知的CostVolume.在DTU和Tanks and Temples数据集上进行实验,结果表明,与主流的多视立体视觉重建方法相比,该方法在三维重建的完整度方面有较大提升,并且拥有良好的泛化性.
