Dense 3D surface reconstruction of large-scale streetscape from vehicle-borne imagery and LiDAR

Accurate and efficient three-dimensional(3D)streetscape reconstruction is the fundamental ability for an exploration vehicle to navigate safely and perform high-level tasks.Recently,remarkable progress has been made in streetscape reconstruction with visual images and light detection and ranging(LiDAR),but they have difficulties either in scaling and reconstructing large-scale outdoors or in efficient processing.To address these issues,this paper proposed an automatic method for incremental dense reconstruction of large-scale 3D streetscapes from coarse to fine at near real time.Firstly,the pose of vehicle is estimated by visual and laser odometry(VLO)and the state-of-the-art pyramid stereo matching network(PSMNet)is introduced to estimate depth information.Then,incremental dense 3D streetscape reconstruction is conducted by key-frame selection and coarse registration with local optimization.Finally,redundant and noise points are removed through multiple filtering,resulting good quality of dense reconstruction.Comprehensive experiments were undertaken to check the visual effect,trajectory pose error and multi-scale model to model cloud comparison(M3C2)based on reference trajectories and reconstructions provided by the state-of-the-art method,showing the precision,recall and F-score of sampling core points(SCPs)are over 80.42%,71.68%and 77.19%,respectively,which verified the proposed method.

Automatic object annotation in streamed and remotely explored large 3D reconstructions

We introduce a novel framework for 3 D scene reconstruction with simultaneous object annotation,using a pre-trained 2 D convolutional neural network(CNN),incremental data streaming,and remote exploration,with a virtual reality setup.It enables versatile integration of any 2 D box detection or segmentation network.We integrate new approaches to(i)asynchronously perform dense 3 D-reconstruction and object annotation at interactive frame rates,(ii)efficiently optimize CNN results in terms of object prediction and spatial accuracy,and(iii)generate computationally-efficient colliders in large triangulated3 D-reconstructions at run-time for 3 D scene interaction.Our method is novel in combining CNNs with long and varying inference time with live 3 D-reconstruction from RGB-D camera input.We further propose a lightweight data structure to store the 3 D-reconstruction data and object annotations to enable fast incremental data transmission for real-time exploration with a remote client,which has not been presented before.Our framework achieves update rates of 22 fps(SSD Mobile Net)and 19 fps(Mask RCNN)for indoor environments up to 800 m^(3).We evaluated the accuracy of 3 D-object detection.Our work provides a versatile foundation for semantic scene understanding of large streamed3 D-reconstructions,while being independent from the CNN’s processing time.Source code is available for non-commercial use.