A Review of RGB-D Camera Calibration Methods

RGB-D camera is a new type of sensor,which can obtain the depth and texture information in an unknown 3D scene simultaneously,and they have been applied in various fields widely.In fact,when implementing such kinds of applications using RGB-D camera,it is necessary to calibrate it first.To the best of our knowledge,at present,there is no existing a systemic summary related to RGB-D camera calibration methods.Therefore,a systemic review of RGB-D camera calibration is concluded as follows.Firstly,the mechanism of obtained measurement and the related principle of RGB-D camera calibration methods are presented.Subsequently,as some specific applications need to fuse depth and color information,the calibration methods of relative pose between depth camera and RGB camera are introduced in Section 2.Then the depth correction models within RGB-D cameras are summarized and compared respectively in Section 3.Thirdly,considering that the angle of the view field of RGB-D camera is smaller and limited to some specific applications,we discuss the calibration models of relative pose among multiple RGB-D cameras in Section 4.At last,the direction and trend of RGB-D camera calibration are prospected and concluded.

Optimized Method for Real-time Texture Reconstruction with RGB-D Camera

With the appearance of RGB-D camera, the field of three-dimensional (3D) reconstruction receives more and more attention. In this paper, we present an optimization approach to produce high-quality textured 3D models based on the real-time 3D reconstruction system. The resulting models of real-time texture reconstruction often suffer from blurring, ghosting, and other artifacts. Our approach addresses this texture quality problem using blur detection and an optimized weight function. Experimental results demonstrate that our approach can improve the quality of textured 3D models by reducing the blur and ghosts on the model surface. © 2017, Tianjin University and Springer-Verlag GmbH Germany.

Semi-Direct Visual Odometry and Mapping System with RGB-D Camera

In this paper a semi-direct visual odometry and mapping system is proposed with a RGB-D camera,which combines the merits of both feature based and direct based methods.The presented system directly estimates the camera motion of two consecutive RGB-D frames by minimizing the photometric error.To permit outliers and noise,a robust sensor model built upon the t-distribution and an error function mixing depth and photometric errors are used to enhance the accuracy and robustness.Local graph optimization based on key frames is used to reduce the accumulative error and refine the local map.The loop closure detection method,which combines the appearance similarity method and spatial location constraints method,increases the speed of detection.Experimental results demonstrate that the proposed approach achieves higher accuracy on the motion estimation and environment reconstruction compared to the other state-of-the-art methods. Moreover,the proposed approach works in real-time on a laptop without a GPU,which makes it attractive for robots equipped with limited computational resources.

Infrastructure-Based Vehicle Localization System for Indoor Parking Lots Using RGB-D Cameras

Accurate vehicle localization is a key technology for autonomous driving tasks in indoor parking lots,such as automated valet parking.Additionally,infrastructure-based cooperative driving systems have become a means to realizing intelligent driving.In this paper,we propose a novel and practical vehicle localization system using infrastructure-based RGB-D cameras for indoor parking lots.In the proposed system,we design a depth data preprocessing method with both simplicity and efficiency to reduce the computational burden resulting from a large amount of data.Meanwhile,the hardware synchronization for all cameras in the sensor network is not implemented owing to the disadvantage that it is extremely cumbersome and would significantly reduce the scalability of our system in mass deployments.Hence,to address the problem of data distortion accompanying vehicle motion,we propose a vehicle localization method by performing template point cloud registration in distributed depth data.Finally,a complete hardware system was built to verify the feasibility of our solution in a real-world environment.Experiments in an indoor parking lot demonstrated the effectiveness and accuracy of the proposed vehicle localization system,with a maximum root mean squared error of 5 cm at 15Hz compared with the ground truth.

基于轻量化目标检测网络的RGB-D视觉SLAM系统

RGB-D SLAM是一种利用深度相机实现同时定位和地图构建的技术。传统的视觉SLAM系统基于对静态环境的假设,然而实际环境中往往存在动态物体,这可能导致SLAM系统的位姿估计出现显著的偏差。针对这一问题,提出了基于轻量化的YOLOv8s目标检测的RGB-D视觉SLAM系统,采用Socket通信方式,将目标检测结果传给SLAM,然后利用Depth Value-RANSAC几何算法剔除检测框内的动态特征点,提高了SLAM系统在动态环境中的定位精度。实验使用TUM数据集进行验证,结果表明,本文系统精度相比ORB-SLAM2有明显提高。与其他SLAM系统相比,本文系统在精度和实时性上有不同程度的改进。

基于RGB-D视觉信息融合的带式输送机煤流量检测方法研究

在煤矿中,精确测量带式输送机煤流量对于煤矿生产和管理至关重要。为了进一步提高通过机器视觉方式检测带式输送机煤流量的精度,提出一种基于RGB-D视觉信息融合的煤流量检测方法,使用RGB-D相机采集煤流信息后,利用RGB图像对深度图像从不同尺度进行增强,运用K-means聚类算法,结合RGB图像对预分割的煤料区域深度图像进一步分割,基于微元法建立煤堆体积计算模型计算煤流量。实验结果表明,该方法的平均检测误差为1.57%,平均耗时263.77 ms,满足实际生产要求,为带式输送机煤流量检测提出了一种新的有效途径。

基于特征调节器和双路径引导的RGB-D室内语义分割

针对室内场景图像语义分割结果不精确、显著图粗糙的问题,提出一种基于多模态特征优化提取和双路径引导解码的网络架构(feature regulator and dual-path guidance,FG-Net)。具体来说,设计的特征调节器对每个阶段的多模态特征依次进行噪声过滤、重加权表示、差异性互补和交互融合,通过强化RGB和深度特征聚合,优化特征提取过程中的多模态特征表示。然后,在解码阶段引入特征交互融合后丰富的跨模态线索,进一步发挥多模态特征的优势。结合双路径协同引导结构,在解码阶段融合多尺度、多层次的特征信息,从而输出更细致的显著图。实验在公开数据集NYUD-v2和SUN RGB-D上进行,在主要评价指标mIoU上达到48.5%,优于其他先进算法。结果表明,该算法实现了更精细的室内场景图像语义分割,表现出了较好的泛化性和鲁棒性。

基于多模态特征交互的RGB-D显著性目标检测

现有的大多数RGB-D显著性目标检测方法利用深度图来提高检测效果,而忽视了其质量的影响。低质量的深度图会对最终显著目标预测结果造成污染,影响显著性检测的性能。为了消除低质量深度图带来的干扰,并准确突出RGB图像中的显著目标,提出了一个用于多模态特征交互的RGB-D显著性目标检测模型。在编码阶段,设计了一个特征交互模块,其包含三个子模块:用于增强特征表述能力的全局特征采集子模块、用于过滤低质量深度信息的深度特征精炼子模块和用于实现特征融合的多模态特征交互子模块。在解码阶段,逐层融合经过特征交互后的多模态特征,实现多层次特征融合。通过在五个基准数据集上与十二种先进方法进行的综合实验表明,该模型在NLPR、SIP和NJU2K数据集上的指标上均优于其他对比方法,其中在NJU2K数据集上,该模型的性能比第二名在平均F值上提升了0.008,加权F值上提升了0.014,E-measure上提升了0.007,表现出了较好的检测效果。

跨模态交互融合与全局感知的RGB-D显著性目标检测

近年来,RGB-D显著性检测方法凭借深度图中丰富的几何结构和空间位置信息,取得了比RGB显著性检测模型更好的性能,受到学术界的高度关注.然而,现有的RGB-D检测模型仍面临着持续提升检测性能的需求.最近兴起的Transformer擅长建模全局信息,而卷积神经网络(CNN)擅长提取局部细节.因此,如何有效结合CNN和Transformer两者的优势,挖掘全局和局部信息,将有助于提升显著性目标检测的精度.为此,提出一种基于跨模态交互融合与全局感知的RGB-D显著性目标检测方法,通过将Transformer网络嵌入U-Net中,从而将全局注意力机制与局部卷积结合在一起,能够更好地对特征进行提取.首先借助U-Net编码-解码结构,高效地提取多层次互补特征并逐级解码生成显著特征图.然后,使用Transformer模块学习高级特征间的全局依赖关系增强特征表示,并针对输入采用渐进上采样融合策略以减少噪声信息的引入.其次,为了减轻低质量深度图带来的负面影响,设计一个跨模态交互融合模块以实现跨模态特征融合.最后,5个基准数据集上的实验结果表明,所提算法与其他最新的算法相比具有显著优势.

面向室内弱纹理场景多特征融合RGB-D SLAM方法

针对室内弱纹理场景下特征点数量不足导致即时定位与建图(SLAM)系统跟踪丢失和重建精度差的问题,提出了一种顾及约束退化的多特征融合RGB-D SLAM算法。为了充分利用线和平面特征对位姿估计的约束,分别建立了线和平面误差方程,并通过对海森矩阵进行特征值分解,定量分析了线和平面特征位姿约束的退化情况,建立了顾及约束退化的多特征融合目标优化函数。此外,基于曼哈顿世界假设,建立了曼哈顿坐标系,充分利用曼哈顿世界假设的优势,对旋转矩阵的“零漂移”进行估计,以提供准确的初始值支持平面匹配和位姿优化。实验结果表明,引入线和面特征建立光束法方程后,所提出的方法在弱纹理数据集ICL-NUIM上的轨迹精度相较于基准的ORB-SLAM2平均提升了37.5%,有效改善了SLAM系统在弱纹理场景中的轨迹精度。

激光雷达与RGB-D相机融合的SLAM建图

对二维激光雷达与RGB-D相机联合标定,采用改进的ORB-SLAM2算法实现稠密的点云地图、八叉树地图、栅格地图的构建。提出了一种将Cartographer算法与改进的ORB-SLAM2算法融合建图的改进算法。实验结果表明,相比传统的ORB-SLAM2算法,改进的融合算法在建图过程中障碍物的识别率达到了96.8%,绝对位姿误差减小了53.2%,提高了建图的精确性和鲁棒性。

基于互学习和促进分割的RGB-D显著性目标检测

RGB-D显著性目标检测主要通过融合RGB图像和深度图(Depth)从给定场景中分割最显著的对象.由于受到原始深度图固有噪声的影响,会导致模型在检测过程中拟入错误的信息.为了改善检测效果,该文提出了一种基于互学习和促进分割的RGB-D显著性目标检测模型,设计一个深度优化模块来获取深度图和预测深度图之间最优的深度信息;引入特征对齐模块和跨模态集成模块完成跨模态的融合;针对分割造成的精度损失问题,构建了一个基于多源特征集成机制的分离重构解码器.在5个公开数据集上进行了实验测试,实验结果表明:所提模型与其他模型相比,准确率更高,网络更加稳定.

基于跨模态特征融合的RGB-D显著性目标检测

RGB-D显著性目标检测因其有效性和易于捕捉深度线索而受到越来越多的关注。现有的工作通常侧重于通过各种融合策略学习共享表示,少有方法明确考虑如何维持RGB和深度的模态特征。提出了一种跨模态特征融合网络,该网络维持RGB-D显著目标检测的RGB和深度的模态,通过探索共享信息以及RGB和深度模态的特性来提高显著检测性能。具体来说,采用RGB模态、深度模态网络和一个共享学习网络来生成RGB和深度模态显著性预测图以及共享显著性预测图。提出了一种跨模态特征融合模块,用于融合共享学习网络中的跨模态特征,然后将这些特征传播到下一层以整合跨层次信息。此外,提出了一种多模态特征聚合模块,将每个单独解码器的模态特定特征整合到共享解码器中,这可以提供丰富的互补多模态信息来提高显著性检测性能。最后,使用跳转连接来组合编码器和解码器层之间的分层特征。通过在4个基准数据集上与7种先进方法进行的实验表明,方法优于其他最先进的方法。

基于级联解码与边缘感知的RGB-D图像语义分割算法

随着深度传感器技术的发展,由于其采样得到的深度图像含有场景的空间几何信息,有助于提升语义分割精度,因此RGB-D图像的语义分割受到了广泛关注,然而现有的基于编码器-解码器架构的双流RGB-D语义分割网络仍存在分割不够精细和边缘区域分割精度低的问题。针对上述问题,论文提出了一种基于级联解码与边缘感知的RGB-D语义分割方法。首先设计了级联解码器,采用自顶向下的解码方式逐级整合多尺度信息,细化分割结果;随后将边缘检测任务以多任务学习的方式引入RGB-D语义分割中,增大卷积神经网络所提特征对边缘像素的关注度,提升分割边缘的精度。在NYUDv2数据集上的实验结果表明,所提出的语义分割方法优于其他对比方法。

Source Camera Identification Algorithm Based on Multi-Scale Feature Fusion

The widespread availability of digital multimedia data has led to a new challenge in digital forensics.Traditional source camera identification algorithms usually rely on various traces in the capturing process.However,these traces have become increasingly difficult to extract due to wide availability of various image processing algorithms.Convolutional Neural Networks(CNN)-based algorithms have demonstrated good discriminative capabilities for different brands and even different models of camera devices.However,their performances is not ideal in case of distinguishing between individual devices of the same model,because cameras of the same model typically use the same optical lens,image sensor,and image processing algorithms,that result in minimal overall differences.In this paper,we propose a camera forensics algorithm based on multi-scale feature fusion to address these issues.The proposed algorithm extracts different local features from feature maps of different scales and then fuses them to obtain a comprehensive feature representation.This representation is then fed into a subsequent camera fingerprint classification network.Building upon the Swin-T network,we utilize Transformer Blocks and Graph Convolutional Network(GCN)modules to fuse multi-scale features from different stages of the backbone network.Furthermore,we conduct experiments on established datasets to demonstrate the feasibility and effectiveness of the proposed approach.

轻量化改进XYZNet的RGB-D特征提取网络

针对用于位姿估计的RGB-D特征提取网络规模过于庞大的问题,提出一种轻量化改进XYZNet的RGB-D特征提取网络。首先设计一种轻量级子网络BaseNet以替换XYZNet中的ResNet18,使得网络规模显著下降的同时获得更强大的性能;然后基于深度可分离卷积设计一种多尺度卷积注意力子模块Rep-MSCA(re-parameterized multi-scale convolutional attention),加强BaseNet提取不同尺度上下文信息的能力,并约束模型的参数量;最后,为了以较小的参数代价提升XYZNet中PointNet的几何特征提取能力,设计一种残差多层感知器模块Rep-ResP(re-parameterized residual multi-layer perceptron)。改进后的网络浮点计算量与参数量分别降低了60.8%和64.8%,推理速度加快了21.2%,在主流数据集LineMOD与YCB-Video上分别取得了0.5%与0.6%的精度提升。改进后的网络更适宜在硬件资源紧张的场景下部署。

Ultrafast pulse-dilation framing camera and its application for time-resolved X-ray diagnostic

An ultrafast framing camera with a pulse-dilation device,a microchannel plate(MCP)imager,and an electronic imaging system were reported.The camera achieved a temporal resolution of 10 ps by using a pulse-dilation device and gated MCP imager,and a spatial resolution of 100μm by using an electronic imaging system comprising combined magnetic lenses.The spatial resolution characteristics of the camera were studied both theoretically and experimentally.The results showed that the camera with combined magnetic lenses reduced the field curvature and acquired a larger working area.A working area with a diameter of 53 mm was created by applying four magnetic lenses to the camera.Furthermore,the camera was used to detect the X-rays produced by the laser-targeting device.The diagnostic results indicated that the width of the X-ray pulse was approximately 18 ps.

Monocular Distance Estimated Based on PTZ Camera

This paper introduces an intelligent computational approach for extracting salient objects fromimages and estimatingtheir distance information with PTZ (Pan-Tilt-Zoom) cameras. PTZ cameras have found wide applications innumerous public places, serving various purposes such as public securitymanagement, natural disastermonitoring,and crisis alarms, particularly with the rapid development of Artificial Intelligence and global infrastructuralprojects. In this paper, we combine Gauss optical principles with the PTZ camera’s capabilities of horizontal andpitch rotation, as well as optical zoom, to estimate the distance of the object.We present a novel monocular objectdistance estimation model based on the Focal Length-Target Pixel Size (FLTPS) relationship, achieving an accuracyrate of over 95% for objects within a 5 km range. The salient object extraction is achieved through a simplifiedconvolution kernel and the utilization of the object’s RGB features, which offer significantly faster computingspeeds compared to Convolutional Neural Networks (CNNs). Additionally, we introduce the dark channel beforethe fog removal algorithm, resulting in a 20 dB increase in image definition, which significantly benefits distanceestimation. Our system offers the advantages of stability and low device load, making it an asset for public securityaffairs and providing a reference point for future developments in surveillance hardware.

Automatic Miscalibration Detection and Correction of LiDAR and Camera Using Motion Cues

This paper aims to develop an automatic miscalibration detection and correction framework to maintain accurate calibration of LiDAR and camera for autonomous vehicle after the sensor drift.First,a monitoring algorithm that can continuously detect the miscalibration in each frame is designed,leveraging the rotational motion each individual sensor observes.Then,as sensor drift occurs,the projection constraints between visual feature points and LiDAR 3-D points are used to compute the scaled camera motion,which is further utilized to align the drifted LiDAR scan with the camera image.Finally,the proposed method is sufficiently compared with two representative approaches in the online experiments with varying levels of random drift,then the method is further extended to the offline calibration experiment and is demonstrated by a comparison with two existing benchmark methods.

面向RGB-D语义分割的多模态任意旋转自监督学习

基于RGB-D数据的自监督学习受到广泛关注,然而大多数方法侧重全局级别的表示学习,会丢失对识别对象至关重要的局部细节信息.由于RGB-D数据中图像和深度具有几何一致性,因此这可以作为线索来指导RGB-D数据的自监督特征表示学习.在本文中,我们提出了ArbRot,它可以无限制地旋转角度并为代理任务生成多个伪标签用于自监督学习,而且还建立了全局和局部之间的上下文联系.本文所提出的ArbRot可以与其他对比学习方法联合训练,构建多模态多代理任务自监督学习框架,以增强图像和深度视图的特征表示一致性,从而为RGB-D语义分割任务提供有效的初始化.在SUN RGB-D和NYU Depth Dataset V2数据集上的实验结果表明,多模态任意旋转自监督学习得到的特征表示质量均高于基线模型.开源代码:https://github.com/Physu/ArbRot.