Dynamic SLAM Visual Odometry Based on Instance Segmentation:A Comprehensive Review

Dynamic Simultaneous Localization and Mapping(SLAM)in visual scenes is currently a major research area in fields such as robot navigation and autonomous driving.However,in the face of complex real-world envi-ronments,current dynamic SLAM systems struggle to achieve precise localization and map construction.With the advancement of deep learning,there has been increasing interest in the development of deep learning-based dynamic SLAM visual odometry in recent years,and more researchers are turning to deep learning techniques to address the challenges of dynamic SLAM.Compared to dynamic SLAM systems based on deep learning methods such as object detection and semantic segmentation,dynamic SLAM systems based on instance segmentation can not only detect dynamic objects in the scene but also distinguish different instances of the same type of object,thereby reducing the impact of dynamic objects on the SLAM system’s positioning.This article not only introduces traditional dynamic SLAM systems based on mathematical models but also provides a comprehensive analysis of existing instance segmentation algorithms and dynamic SLAM systems based on instance segmentation,comparing and summarizing their advantages and disadvantages.Through comparisons on datasets,it is found that instance segmentation-based methods have significant advantages in accuracy and robustness in dynamic environments.However,the real-time performance of instance segmentation algorithms hinders the widespread application of dynamic SLAM systems.In recent years,the rapid development of single-stage instance segmentationmethods has brought hope for the widespread application of dynamic SLAM systems based on instance segmentation.Finally,possible future research directions and improvementmeasures are discussed for reference by relevant professionals.

Semi-Direct Visual Odometry and Mapping System with RGB-D Camera

In this paper a semi-direct visual odometry and mapping system is proposed with a RGB-D camera,which combines the merits of both feature based and direct based methods.The presented system directly estimates the camera motion of two consecutive RGB-D frames by minimizing the photometric error.To permit outliers and noise,a robust sensor model built upon the t-distribution and an error function mixing depth and photometric errors are used to enhance the accuracy and robustness.Local graph optimization based on key frames is used to reduce the accumulative error and refine the local map.The loop closure detection method,which combines the appearance similarity method and spatial location constraints method,increases the speed of detection.Experimental results demonstrate that the proposed approach achieves higher accuracy on the motion estimation and environment reconstruction compared to the other state-of-the-art methods. Moreover,the proposed approach works in real-time on a laptop without a GPU,which makes it attractive for robots equipped with limited computational resources.

A robust RGB-D visual odometry with moving object detection in dynamic indoor scenes

Simultaneous localisation and mapping(SLAM)are the basis for many robotic applications.As the front end of SLAM,visual odometry is mainly used to estimate camera pose.In dynamic scenes,classical methods are deteriorated by dynamic objects and cannot achieve satisfactory results.In order to improve the robustness of visual odometry in dynamic scenes,this paper proposed a dynamic region detection method based on RGBD images.Firstly,all feature points on the RGB image are classified as dynamic and static using a triangle constraint and the epipolar geometric constraint successively.Meanwhile,the depth image is clustered using the K-Means method.The classified feature points are mapped to the clustered depth image,and a dynamic or static label is assigned to each cluster according to the number of dynamic feature points.Subsequently,a dynamic region mask for the RGB image is generated based on the dynamic clusters in the depth image,and the feature points covered by the mask are all removed.The remaining static feature points are applied to estimate the camera pose.Finally,some experimental results are provided to demonstrate the feasibility and performance.

一种室内环境下点线特征综合的RGB-D VO算法

针对弱纹理场景下点特征提取不足影响同步定位与建图(SLAM)算法定位精度的问题,提出了一种室内环境下点线特征综合的RGB-D视觉里程计(VO)算法。通过跟踪深度信息计算垂直主导方向,基于曼哈顿假设使用线特征来加权搜索两个水平自由度,提取并优化曼哈顿坐标系;综合场景的结构规律,与点线特征的重投影误差进行联合优化,同时在位姿估计和局部地图优化中对残差引入自适应权重,提高位姿估计精度。实验结果表明,所提算法在ICL-NUIM数据集中的绝对轨迹均方根误差相比于ORB-SLAM2和MSC-VO分别平均减少62.93%、37.04%,与Planar-SLAM和Manhattan-SLAM精度相当;在TUM数据集中相比于ORB-SLAM2、Planar-SLAM、MSC-VO和Manhattan-SLAM,绝对轨迹均方根误差分别平均减小21.43%、54.40%、35.08%和26.94%;在TAMU数据集中相比于ORB-SLAM2,回环漂移平均减小43.34%。

Overfitting Reduction of Pose Estimation for Deep Learning Visual Odometry

Error or drift is frequently produced in pose estimation based on geometric"feature detection and tracking"monocular visual odometry(VO)when the speed of camera movement exceeds 1.5 m/s.While,in most VO methods based on deep learning,weight factors are in the form of fixed values,which are easy to lead to overfitting.A new measurement system,for monocular visual odometry,named Deep Learning Visual Odometry(DLVO),is proposed based on neural network.In this system,Convolutional Neural Network(CNN)is used to extract feature and perform feature matching.Moreover,Recurrent Neural Network(RNN)is used for sequence modeling to estimate camera’s 6-dof poses.Instead of fixed weight values of CNN,Bayesian distribution of weight factors are introduced in order to effectively solve the problem of network overfitting.The 18,726 frame images in KITTI dataset are used for training network.This system can increase the generalization ability of network model in prediction process.Compared with original Recurrent Convolutional Neural Network(RCNN),our method can reduce the loss of test model by 5.33%.And it’s an effective method in improving the robustness of translation and rotation information than traditional VO methods.

An RGB-D Camera Based Visual Positioning System for Assistive Navigation by a Robotic Navigation Aid

There are about 253 million people with visual impairment worldwide.Many of them use a white cane and/or a guide dog as the mobility tool for daily travel.Despite decades of efforts,electronic navigation aid that can replace white cane is still research in progress.In this paper,we propose an RGB-D camera based visual positioning system(VPS)for real-time localization of a robotic navigation aid(RNA)in an architectural floor plan for assistive navigation.The core of the system is the combination of a new 6-DOF depth-enhanced visual-inertial odometry(DVIO)method and a particle filter localization(PFL)method.DVIO estimates RNA’s pose by using the data from an RGB-D camera and an inertial measurement unit(IMU).It extracts the floor plane from the camera’s depth data and tightly couples the floor plane,the visual features(with and without depth data),and the IMU’s inertial data in a graph optimization framework to estimate the device’s 6-DOF pose.Due to the use of the floor plane and depth data from the RGB-D camera,DVIO has a better pose estimation accuracy than the conventional VIO method.To reduce the accumulated pose error of DVIO for navigation in a large indoor space,we developed the PFL method to locate RNA in the floor plan.PFL leverages geometric information of the architectural CAD drawing of an indoor space to further reduce the error of the DVIO-estimated pose.Based on VPS,an assistive navigation system is developed for the RNA prototype to assist a visually impaired person in navigating a large indoor space.Experimental results demonstrate that:1)DVIO method achieves better pose estimation accuracy than the state-of-the-art VIO method and performs real-time pose estimation(18 Hz pose update rate)on a UP Board computer;2)PFL reduces the DVIO-accrued pose error by 82.5%on average and allows for accurate wayfinding(endpoint position error≤45 cm)in large indoor spaces.

Science Letters:Visual odometry for road vehicles—feasibility analysis

Estimating the global position of a road vehicle without using GPS is a challenge that many scientists look forward to solving in the near future. Normally, inertial and odometry sensors are used to complement GPS measures in an attempt to provide a means for maintaining vehicle odometry during GPS outage. Nonetheless, recent experiments have demonstrated that computer vision can also be used as a valuable source to provide what can be denoted as visual odometry. For this purpose, vehicle motion can be estimated using a non-linear, photogrametric approach based on RAndom SAmple Consensus (RANSAC). The results prove that the detection and selection of relevant feature points is a crucial factor in the global performance of the visual odometry algorithm. The key issues for further improvement are discussed in this letter.

Human Visual Attention Mechanism-Inspired Point-and-Line Stereo Visual Odometry for Environments with Uneven Distributed Features

Visual odometry is critical in visual simultaneous localization and mapping for robot navigation.However,the pose estimation performance of most current visual odometry algorithms degrades in scenes with unevenly distributed features because dense features occupy excessive weight.Herein,a new human visual attention mechanism for point-and-line stereo visual odometry,which is called point-line-weight-mechanism visual odometry(PLWM-VO),is proposed to describe scene features in a global and balanced manner.A weight-adaptive model based on region partition and region growth is generated for the human visual attention mechanism,where sufficient attention is assigned to position-distinctive objects(sparse features in the environment).Furthermore,the sum of absolute differences algorithm is used to improve the accuracy of initialization for line features.Compared with the state-of-the-art method(ORB-VO),PLWM-VO show a 36.79%reduction in the absolute trajectory error on the Kitti and Euroc datasets.Although the time consumption of PLWM-VO is higher than that of ORB-VO,online test results indicate that PLWM-VO satisfies the real-time demand.The proposed algorithm not only significantly promotes the environmental adaptability of visual odometry,but also quantitatively demonstrates the superiority of the human visual attention mechanism.

A Study on Planetary Visual Odometry Optimization： Time Constraints and Reliability

Robust and efficient vision systems are essential in such a way to support different kinds of autonomous robotic behaviors linked to the capability to interact with the surrounding environment, without relying on any a priori knowledge. Within space missions, above all those involving rovers that have to explore planetary surfaces, vision can play a key role in the improvement of autonomous navigation functionalities： besides obstacle avoidance and hazard detection along the traveling, vision can in fact provide accurate motion estimation in order to constantly monitor all paths executed by the rover. The present work basically regards the development of an effective visual odometry system, focusing as much as possible on issues such as continuous operating mode, system speed and reliability.

基于RGB-D相机的视觉里程计实现

针对移动机器人在未知环境中的自身定位问题,提出了一种基于RGB-D相机的移动机器人运动轨迹估计方法.首先,提取当前图像的ORB特征并与关键帧进行特征匹配;然后,采用结合特征匹配质量和深度信息的PROSAC算法对帧间运动进行迭代估计;最后,提取关键帧并利用g2o求解器进行局部优化,得到关键帧位姿的最优估计,进而得到机器人的运动轨迹.实验结果表明:与RANSAC+ICP算法相比,该方法能有效提高移动机器人的定位精度.

基于RGB-D相机的室内环境3D地图创建

RGB-D相机(如微软的Kinect)能够在获取彩色图像的同时得到每个像素的深度信息,在移动机器人三维地图创建方向具有广泛应用。设计了一种利用RGB-D相机进行机器人自定位及创建室内场景三维模型的方法,该方法由RGB-D相机获取周围环境的连续帧信息;提取并匹配连续帧间的SURF特征点,通过特征点的位置变化计算机器人的位姿并结合非线性最小二乘优化算法最小化对应点的双向投影误差;结合关键帧技术及观察中心法将相机观测到的三维点云依据当前位姿投影到全局地图。选择三个不同的场景实验了该方法,并对比了不同特征点下该方法的效果,方法在轨迹长度为5.88 m情况下误差仅为0.023 m,能够准确地创建周围环境的三维模型。

基于RGB-D图像的三维同步定位与建图研究

针对移动机器人三维同步定位与建图过程中机器人位姿误差累积问题,提出了一种位姿全局优化方法提高机器人定位精度和建图质量。该方法在帧到帧配准模型的视觉里程计的基础上,通过基于图像匹配的闭环检测来增加机器人位姿间的约束,在构建位姿图过程中采用局部回环结合随机大回环策略提高位姿优化效率,最后采用g2o(general graph optimization)算法对机器人位姿进行全局优化。此外,提出了一种关键帧选取方法,以减少系统计算资源及内存空间的消耗。实验结果表明,该方法在运动轨迹3.96 m的情况下均方根误差仅为8.7 mm,并能准确构建出室内场景的三维地图。

一种基于线特征的RGB-D视觉里程计算法

基于特征点的视觉里程计在点特征稀少的环境下难以得到足够的匹配点对,从而导致相机运动估计失败,因而提出采集人造环境中特征明显的边缘作为线特征来提高视觉里程计算法的稳定性。采用深度相机获取的RGB图像进行LSD线特征提取,推断线特征对应的图像位置的深度信息,避免深度缺失,将线段上的2D点反投影为3D点,拟合3D点为三维直线,利用线特征匹配关系进行位姿估计。此外在位姿优化部分进行改进,利用拟合直线过程中的最佳过点,以及重投影的线段与观测线段的角度误差信息,推导了误差关于位姿扰动的雅克比矩阵,在图优化时利用重投影误差优化相机位姿,拓展了传统的优化方法。基于TUM缺少点特征的数据集的实验结果表明所提出的线特征视觉里程计方法相比ORB-SLAM2的轨迹估计精度提高63%,并能完整地跟踪轨迹。实验结果表明所提出算法在欠特征点环境中表现出了较高的精度和稳定性。

基于RGB-D摄像机的室内三维彩色点云地图构建

创建室内环境的三维地图可以使用RGB-D摄相机获取多帧率的彩色图和深度图,结合视觉SLAM算法,完成彩信息与深度信息的结合并构建真实三维环境地图数据是常用的一种方法.通过Kinect传感器采集室内环境的RGB图和Depth图,改进了ORB算法并进行特征提取和基于FLANN搜索的图像匹配算法匹配图像特征,使用PNPRANSAC方法估计图像的运动,为抑制在点云配准过程中累积误差造成的位姿漂移采用g2o图优化,实现探索室内未知环境的三维彩色点云空间,完成周围环境地图的创建.实验结果表明,以Kinect传感器实现视觉SLAM的方法对于室内三维点云数据创建有较好的效果,具有一定的理论和实际应用价值.

基于RGB-D视觉里程计估计算法的研究

研究了基于ORB的特征点法和半稠密直接法在视觉特征丰富以及视觉特征缺乏环境当中的实时性、精确性和鲁棒性。实验结果表明,基于ORB的特征点法在视觉特征丰富的环境中比半稠密直接法的实时性和精确性好,而在视觉特征缺乏的环境中,半稠密直接法的鲁棒性更好。

Visual SLAM Based on Object Detection Network:A Review

Visual simultaneous localization and mapping(SLAM)is crucial in robotics and autonomous driving.However,traditional visual SLAM faces challenges in dynamic environments.To address this issue,researchers have proposed semantic SLAM,which combines object detection,semantic segmentation,instance segmentation,and visual SLAM.Despite the growing body of literature on semantic SLAM,there is currently a lack of comprehensive research on the integration of object detection and visual SLAM.Therefore,this study aims to gather information from multiple databases and review relevant literature using specific keywords.It focuses on visual SLAM based on object detection,covering different aspects.Firstly,it discusses the current research status and challenges in this field,highlighting methods for incorporating semantic information from object detection networks into mileage measurement,closed-loop detection,and map construction.It also compares the characteristics and performance of various visual SLAM object detection algorithms.Lastly,it provides an outlook on future research directions and emerging trends in visual SLAM.Research has shown that visual SLAM based on object detection has significant improvements compared to traditional SLAM in dynamic point removal,data association,point cloud segmentation,and other technologies.It can improve the robustness and accuracy of the entire SLAM system and can run in real time.With the continuous optimization of algorithms and the improvement of hardware level,object visual SLAM has great potential for development.

PLVO:基于平面和直线融合的RGB-D视觉里程计

针对利用平面特征计算RGB-D相机位姿时的求解退化问题,提出平面和直线融合的RGB-D视觉里程计(Plane-line-based RGB-D visual odometry,PLVO).首先,提出基于平面-直线混合关联图(Plane-line hybrid association graph,PLHAG)的多特征关联方法,充分考虑平面和平面、平面和直线之间的几何关系,对平面和直线两类几何特征进行一体化关联.然后,提出基于平面和直线主辅相济、自适应融合的RGB-D相机位姿估计方法.具体来说,鉴于平面特征通常比直线特征具有更好的准确性和稳定性,通过自适应加权的方法,确保平面特征在位姿计算中的主导作用,而对平面特征无法约束的位姿自由度(Degree of freedom,DoF),使用直线特征进行补充,得到相机的6自由度位姿估计结果,从而实现两类特征的融合,解决了单纯使用平面特征求解位姿时的退化问题.最后,通过公开数据集上的定量实验以及真实室内环境下的机器人实验,验证了所提出方法的有效性.

Unsupervised Domain Adaptation Learning Algorithm for RGB-D Stairway Recognition

Detection and recognition of a stairway as upstairs,downstairs and negative(e.g.,ladder,level ground)are the fundamentals of assisting the visually impaired to travel independently in unfamiliar environments.Previous studies have focused on using massive amounts of RGB-D scene data to train traditional machine learning(ML)-based models to detect and recognize stationary stairway and escalator stairway separately.Nevertheless,none of them consider jointly training these two similar but different datasets to achieve better performance.This paper applies an adversarial learning algorithm on the indicated unsupervised domain adaptation scenario to transfer knowledge learned from the labeled RGB-D escalator stairway dataset to the unlabeled RGB-D stationary dataset.By utilizing the developed method,a feedforward convolutional neural network(CNN)-based feature extractor with five convolution layers can achieve 100%classification accuracy on testing the labeled escalator stairway data distributions and 80.6%classification accuracy on testing the unlabeled stationary data distributions.The success of the developed approach is demonstrated for classifying stairway on these two domains with a limited amount of data.To further demonstrate the effectiveness of the proposed method,the same CNN model is evaluated without domain adaptation and the results are compared with those of the presented architecture.

移动机器人视觉里程计技术研究综述

随着移动机器人技术不断发展,里程计技术已经成为移动机器人实现环境感知的关键技术,其发展水平对提高机器人的自主化和智能化具有重要意义。首先,系统阐述了同步定位与地图构建(Simultaneous localization and mapping,SLAM)中激光SLAM和视觉SLAM的发展近况,阐述了经典SLAM框架及其数学描述,简要介绍了3类常见相机的相机模型及其视觉里程计的数学描述。其次,分别对传统视觉里程计和深度学习里程计的研究进展进行系统阐述。对比分析了近10年来各类里程计算法的优势与不足。另外,对比分析了7种常用数据集的性能。最后,从精度、鲁棒性、数据集、多模态等方面总结了里程计技术面临的问题,从提高算法实时性、鲁棒性等方面展望了视觉里程计的发展趋势为:更加智能化、小型化新型传感器的发展;与无监督学习融合;语义表达技术的提高;集群机器人协同技术的发展。

融合点、面特征的RGB-D视觉里程计

基于ICP的视觉里程计算法存在计算量大、收敛缓慢的问题,并且室内环境存在大量结构化平面。为此,提出了一种融合点特征和面特征的RGB-D视觉里程计算法。首先利用改进后的ORB算法提取环境中的角点信息,然后利用层次聚类和主成分分析进行平面拟合,提取出环境中的平面特征,接下来根据特征匹配对,使用融合了点面特征的特征匹配算法解算出相机的位姿估计,最后再利用ICP算法精确计算相机的位姿变化。方法再TUM数据集中表现出了良好的性能,效果显著优于传统的ElastcFusion算法,重建结果的表面信息更为丰富准确,累积误差的精度提升,满足实时重建的要求,具有一定的实用价值。