基于局部对齐单目视频深度的三维场景重建

单目深度估计方法在各种场景下已经取得了较强的鲁棒性,该类方法通常预测尺度偏移量未知的不变仿射深度而非度量深度,因为收集大规模的不变仿射深度训练数据比收集度量深度训练数据容易得多。然而,在某些基于视频的应用场景中,例如视频深度估计和三维场景重建,每帧预测的深度中存在的未知比例和偏移量值可能会导致预测的深度不一致。为了解决该问题,我们提出了一种基于局部加权线性回归的方法,通过利用稀疏锚点恢复深度的尺度图和偏移量图,以保证连续帧之间的一致性。大量的实验表明,我们的方法可以在几个零样本基准上显著降低现有技术方法的Rel误差(相对误差)。此外,我们收集了630万张RGBD图像对来训练鲁棒的深度模型。通过局部恢复尺度和偏移量,我们的ResNet50-backbone模型性能甚至超过了最先进的DPT ViT-Large模型。与基于几何的重建方法相结合,我们提出了一种新的稠密三维场景重建流程,既能受益于稀疏点的尺度一致性,又能受益于单目深度估计方法的鲁棒性。通过对视频的每一帧依次预测深度图,我们可以重建出准确的三维场景几何信息。

Towards Domain-agnostic Depth Completion

Existing depth completion methods are often targeted at a specific sparse depth type and generalize poorly across task domains.We present a method to complete sparse/semi-dense,noisy,and potentially low-resolution depth maps obtained by various range sensors,including those in modern mobile phones,or by multi-view reconstruction algorithms.Our method leverages a data-driven prior in the form of a single image depth prediction network trained on large-scale datasets,the output of which is used as an input to our model.We propose an effective training scheme where we simulate various sparsity patterns in typical task domains.In addition,we design two new benchmarks to evaluate the generalizability and robustness of depth completion methods.Our simple method shows superior cross-domain generalization ability against state-of-the-art depth completion methods,introducing a practical solution to highqualitydepthcapture onamobile device.