Infrastructure-Based Vehicle Localization System for Indoor Parking Lots Using RGB-D Cameras

Accurate vehicle localization is a key technology for autonomous driving tasks in indoor parking lots,such as automated valet parking.Additionally,infrastructure-based cooperative driving systems have become a means to realizing intelligent driving.In this paper,we propose a novel and practical vehicle localization system using infrastructure-based RGB-D cameras for indoor parking lots.In the proposed system,we design a depth data preprocessing method with both simplicity and efficiency to reduce the computational burden resulting from a large amount of data.Meanwhile,the hardware synchronization for all cameras in the sensor network is not implemented owing to the disadvantage that it is extremely cumbersome and would significantly reduce the scalability of our system in mass deployments.Hence,to address the problem of data distortion accompanying vehicle motion,we propose a vehicle localization method by performing template point cloud registration in distributed depth data.Finally,a complete hardware system was built to verify the feasibility of our solution in a real-world environment.Experiments in an indoor parking lot demonstrated the effectiveness and accuracy of the proposed vehicle localization system,with a maximum root mean squared error of 5 cm at 15Hz compared with the ground truth.

FusionMLS:Highly dynamic 3D reconstruction with consumergrade RGB-D cameras

Multi-view dynamic three-dimensional reconstruction has typically required the use of custom shutter-synchronized camera rigs in order to capture scenes containing rapid movements or complex topology changes. In this paper, we demonstrate that multiple unsynchronized low-cost RGB-D cameras can be used for the same purpose. To alleviate issues caused by unsynchronized shutters, we propose a novel depth frame interpolation technique that allows synchronized data capture from highly dynamic 3 D scenes. To manage the resulting huge number of input depth images, we also introduce an efficient moving least squares-based volumetric reconstruction method that generates triangle meshes of the scene. Our approach does not store the reconstruction volume in memory,making it memory-efficient and scalable to large scenes.Our implementation is completely GPU based and works in real time. The results shown herein, obtained with real data, demonstrate the effectiveness of our proposed method and its advantages compared to stateof-the-art approaches.

基于RGB-D视觉信息融合的带式输送机煤流量检测方法研究

在煤矿中,精确测量带式输送机煤流量对于煤矿生产和管理至关重要。为了进一步提高通过机器视觉方式检测带式输送机煤流量的精度,提出一种基于RGB-D视觉信息融合的煤流量检测方法,使用RGB-D相机采集煤流信息后,利用RGB图像对深度图像从不同尺度进行增强,运用K-means聚类算法,结合RGB图像对预分割的煤料区域深度图像进一步分割,基于微元法建立煤堆体积计算模型计算煤流量。实验结果表明,该方法的平均检测误差为1.57%,平均耗时263.77 ms,满足实际生产要求,为带式输送机煤流量检测提出了一种新的有效途径。

基于特征调节器和双路径引导的RGB-D室内语义分割

针对室内场景图像语义分割结果不精确、显著图粗糙的问题,提出一种基于多模态特征优化提取和双路径引导解码的网络架构(feature regulator and dual-path guidance,FG-Net)。具体来说,设计的特征调节器对每个阶段的多模态特征依次进行噪声过滤、重加权表示、差异性互补和交互融合,通过强化RGB和深度特征聚合,优化特征提取过程中的多模态特征表示。然后,在解码阶段引入特征交互融合后丰富的跨模态线索,进一步发挥多模态特征的优势。结合双路径协同引导结构,在解码阶段融合多尺度、多层次的特征信息,从而输出更细致的显著图。实验在公开数据集NYUD-v2和SUN RGB-D上进行,在主要评价指标mIoU上达到48.5%,优于其他先进算法。结果表明,该算法实现了更精细的室内场景图像语义分割,表现出了较好的泛化性和鲁棒性。

跨模态交互融合与全局感知的RGB-D显著性目标检测

近年来,RGB-D显著性检测方法凭借深度图中丰富的几何结构和空间位置信息,取得了比RGB显著性检测模型更好的性能,受到学术界的高度关注.然而,现有的RGB-D检测模型仍面临着持续提升检测性能的需求.最近兴起的Transformer擅长建模全局信息,而卷积神经网络(CNN)擅长提取局部细节.因此,如何有效结合CNN和Transformer两者的优势,挖掘全局和局部信息,将有助于提升显著性目标检测的精度.为此,提出一种基于跨模态交互融合与全局感知的RGB-D显著性目标检测方法,通过将Transformer网络嵌入U-Net中,从而将全局注意力机制与局部卷积结合在一起,能够更好地对特征进行提取.首先借助U-Net编码-解码结构,高效地提取多层次互补特征并逐级解码生成显著特征图.然后,使用Transformer模块学习高级特征间的全局依赖关系增强特征表示,并针对输入采用渐进上采样融合策略以减少噪声信息的引入.其次,为了减轻低质量深度图带来的负面影响,设计一个跨模态交互融合模块以实现跨模态特征融合.最后,5个基准数据集上的实验结果表明,所提算法与其他最新的算法相比具有显著优势.

Study on Color Information Degradation Induced byγ-ray Radiation in CMOS Cameras

Theγ-rays are widely and abundantly present in strong nuclear radiation environments,and when they act on the camera equipment used to obtain environmental visual information on nuclear robots,radiation effects will occur,which will degrade the performance of the camera system,reduce the imaging quality,and even cause catastrophic consequences.Color reducibility is an important index for evaluating the imaging quality of color camera,but its degradation mechanism in a nuclear radiation environment is still unclear.In this paper,theγ-ray irradiation experiments of CMOS cameras were carried out to analyse the degradation law of the camera’s color reducibility with cumulative irradiation and reveal the degradation mechanism of the color information of the CMOS camera underγ-ray irradiation.The results show that the spectral response of CMOS image sensor(CIS)and the spectral transmittance of lens after irradiation affect the values of a^(*)and b^(*)in the LAB color model.While the full well capacity(FWC)of CIS and transmittance of lens affect the value of L^(*)in the LAB color model,thus increase color difference and reduce brightness,the combined effect of color difference and brightness degradation will reduce the color reducibility of CMOS cameras.Therefore,the degradation of the color information of the CMOS camera afterγ-ray irradiation mainly comes from the changes in the FWC and spectral response of CIS,and the spectral transmittance of lens.

High-quality indoor scene 3D reconstruction with RGB-D cameras:A brief review

High-quality 3D reconstruction is an important topic in computer graphics and computer vision with many applications,such as robotics and augmented reality.The advent of consumer RGB-D cameras has made a profound advance in indoor scene reconstruction.For the past few years,researchers have spent significant effort to develop algorithms to capture 3D models with RGB-D cameras.As depth images produced by consumer RGB-D cameras are noisy and incomplete when surfaces are shiny,bright,transparent,or far from the camera,obtaining highquality 3D scene models is still a challenge for existing systems.We here review high-quality 3D indoor scene reconstruction methods using consumer RGB-D cameras.In this paper,we make comparisons and analyses from the following aspects:(i)depth processing methods in 3D reconstruction are reviewed in terms of enhancement and completion,(ii)ICP-based,feature-based,and hybrid methods of camera pose estimation methods are reviewed,and(iii)surface reconstruction methods are reviewed in terms of surface fusion,optimization,and completion.The performance of state-of-the-art methods is also compared and analyzed.This survey will be useful for researchers who want to follow best practices in designing new high-quality 3D reconstruction methods.

基于互学习和促进分割的RGB-D显著性目标检测

RGB-D显著性目标检测主要通过融合RGB图像和深度图(Depth)从给定场景中分割最显著的对象.由于受到原始深度图固有噪声的影响,会导致模型在检测过程中拟入错误的信息.为了改善检测效果,该文提出了一种基于互学习和促进分割的RGB-D显著性目标检测模型,设计一个深度优化模块来获取深度图和预测深度图之间最优的深度信息;引入特征对齐模块和跨模态集成模块完成跨模态的融合;针对分割造成的精度损失问题,构建了一个基于多源特征集成机制的分离重构解码器.在5个公开数据集上进行了实验测试,实验结果表明:所提模型与其他模型相比,准确率更高,网络更加稳定.

基于跨模态特征融合的RGB-D显著性目标检测

RGB-D显著性目标检测因其有效性和易于捕捉深度线索而受到越来越多的关注。现有的工作通常侧重于通过各种融合策略学习共享表示,少有方法明确考虑如何维持RGB和深度的模态特征。提出了一种跨模态特征融合网络,该网络维持RGB-D显著目标检测的RGB和深度的模态,通过探索共享信息以及RGB和深度模态的特性来提高显著检测性能。具体来说,采用RGB模态、深度模态网络和一个共享学习网络来生成RGB和深度模态显著性预测图以及共享显著性预测图。提出了一种跨模态特征融合模块,用于融合共享学习网络中的跨模态特征,然后将这些特征传播到下一层以整合跨层次信息。此外,提出了一种多模态特征聚合模块,将每个单独解码器的模态特定特征整合到共享解码器中,这可以提供丰富的互补多模态信息来提高显著性检测性能。最后,使用跳转连接来组合编码器和解码器层之间的分层特征。通过在4个基准数据集上与7种先进方法进行的实验表明,方法优于其他最先进的方法。

Source Camera Identification Algorithm Based on Multi-Scale Feature Fusion

The widespread availability of digital multimedia data has led to a new challenge in digital forensics.Traditional source camera identification algorithms usually rely on various traces in the capturing process.However,these traces have become increasingly difficult to extract due to wide availability of various image processing algorithms.Convolutional Neural Networks(CNN)-based algorithms have demonstrated good discriminative capabilities for different brands and even different models of camera devices.However,their performances is not ideal in case of distinguishing between individual devices of the same model,because cameras of the same model typically use the same optical lens,image sensor,and image processing algorithms,that result in minimal overall differences.In this paper,we propose a camera forensics algorithm based on multi-scale feature fusion to address these issues.The proposed algorithm extracts different local features from feature maps of different scales and then fuses them to obtain a comprehensive feature representation.This representation is then fed into a subsequent camera fingerprint classification network.Building upon the Swin-T network,we utilize Transformer Blocks and Graph Convolutional Network(GCN)modules to fuse multi-scale features from different stages of the backbone network.Furthermore,we conduct experiments on established datasets to demonstrate the feasibility and effectiveness of the proposed approach.

面向室内弱纹理场景多特征融合RGB-D SLAM方法

针对室内弱纹理场景下特征点数量不足导致即时定位与建图(SLAM)系统跟踪丢失和重建精度差的问题,提出了一种顾及约束退化的多特征融合RGB-D SLAM算法。为了充分利用线和平面特征对位姿估计的约束,分别建立了线和平面误差方程,并通过对海森矩阵进行特征值分解,定量分析了线和平面特征位姿约束的退化情况,建立了顾及约束退化的多特征融合目标优化函数。此外,基于曼哈顿世界假设,建立了曼哈顿坐标系,充分利用曼哈顿世界假设的优势,对旋转矩阵的“零漂移”进行估计,以提供准确的初始值支持平面匹配和位姿优化。实验结果表明,引入线和面特征建立光束法方程后,所提出的方法在弱纹理数据集ICL-NUIM上的轨迹精度相较于基准的ORB-SLAM2平均提升了37.5%,有效改善了SLAM系统在弱纹理场景中的轨迹精度。

激光雷达与RGB-D相机融合的SLAM建图

对二维激光雷达与RGB-D相机联合标定,采用改进的ORB-SLAM2算法实现稠密的点云地图、八叉树地图、栅格地图的构建。提出了一种将Cartographer算法与改进的ORB-SLAM2算法融合建图的改进算法。实验结果表明,相比传统的ORB-SLAM2算法,改进的融合算法在建图过程中障碍物的识别率达到了96.8%,绝对位姿误差减小了53.2%,提高了建图的精确性和鲁棒性。

轻量化改进XYZNet的RGB-D特征提取网络

针对用于位姿估计的RGB-D特征提取网络规模过于庞大的问题,提出一种轻量化改进XYZNet的RGB-D特征提取网络。首先设计一种轻量级子网络BaseNet以替换XYZNet中的ResNet18,使得网络规模显著下降的同时获得更强大的性能;然后基于深度可分离卷积设计一种多尺度卷积注意力子模块Rep-MSCA(re-parameterized multi-scale convolutional attention),加强BaseNet提取不同尺度上下文信息的能力,并约束模型的参数量;最后,为了以较小的参数代价提升XYZNet中PointNet的几何特征提取能力,设计一种残差多层感知器模块Rep-ResP(re-parameterized residual multi-layer perceptron)。改进后的网络浮点计算量与参数量分别降低了60.8%和64.8%,推理速度加快了21.2%,在主流数据集LineMOD与YCB-Video上分别取得了0.5%与0.6%的精度提升。改进后的网络更适宜在硬件资源紧张的场景下部署。

Ultrafast pulse-dilation framing camera and its application for time-resolved X-ray diagnostic

An ultrafast framing camera with a pulse-dilation device,a microchannel plate(MCP)imager,and an electronic imaging system were reported.The camera achieved a temporal resolution of 10 ps by using a pulse-dilation device and gated MCP imager,and a spatial resolution of 100μm by using an electronic imaging system comprising combined magnetic lenses.The spatial resolution characteristics of the camera were studied both theoretically and experimentally.The results showed that the camera with combined magnetic lenses reduced the field curvature and acquired a larger working area.A working area with a diameter of 53 mm was created by applying four magnetic lenses to the camera.Furthermore,the camera was used to detect the X-rays produced by the laser-targeting device.The diagnostic results indicated that the width of the X-ray pulse was approximately 18 ps.

Monocular Distance Estimated Based on PTZ Camera

This paper introduces an intelligent computational approach for extracting salient objects fromimages and estimatingtheir distance information with PTZ (Pan-Tilt-Zoom) cameras. PTZ cameras have found wide applications innumerous public places, serving various purposes such as public securitymanagement, natural disastermonitoring,and crisis alarms, particularly with the rapid development of Artificial Intelligence and global infrastructuralprojects. In this paper, we combine Gauss optical principles with the PTZ camera’s capabilities of horizontal andpitch rotation, as well as optical zoom, to estimate the distance of the object.We present a novel monocular objectdistance estimation model based on the Focal Length-Target Pixel Size (FLTPS) relationship, achieving an accuracyrate of over 95% for objects within a 5 km range. The salient object extraction is achieved through a simplifiedconvolution kernel and the utilization of the object’s RGB features, which offer significantly faster computingspeeds compared to Convolutional Neural Networks (CNNs). Additionally, we introduce the dark channel beforethe fog removal algorithm, resulting in a 20 dB increase in image definition, which significantly benefits distanceestimation. Our system offers the advantages of stability and low device load, making it an asset for public securityaffairs and providing a reference point for future developments in surveillance hardware.

基于多模态特征交互的RGB-D显著性目标检测

现有的大多数RGB-D显著性目标检测方法利用深度图来提高检测效果,而忽视了其质量的影响。低质量的深度图会对最终显著目标预测结果造成污染,影响显著性检测的性能。为了消除低质量深度图带来的干扰,并准确突出RGB图像中的显著目标,提出了一个用于多模态特征交互的RGB-D显著性目标检测模型。在编码阶段,设计了一个特征交互模块,其包含三个子模块:用于增强特征表述能力的全局特征采集子模块、用于过滤低质量深度信息的深度特征精炼子模块和用于实现特征融合的多模态特征交互子模块。在解码阶段,逐层融合经过特征交互后的多模态特征,实现多层次特征融合。通过在五个基准数据集上与十二种先进方法进行的综合实验表明,该模型在NLPR、SIP和NJU2K数据集上的指标上均优于其他对比方法,其中在NJU2K数据集上,该模型的性能比第二名在平均F值上提升了0.008,加权F值上提升了0.014,E-measure上提升了0.007,表现出了较好的检测效果。

Automatic Miscalibration Detection and Correction of LiDAR and Camera Using Motion Cues

This paper aims to develop an automatic miscalibration detection and correction framework to maintain accurate calibration of LiDAR and camera for autonomous vehicle after the sensor drift.First,a monitoring algorithm that can continuously detect the miscalibration in each frame is designed,leveraging the rotational motion each individual sensor observes.Then,as sensor drift occurs,the projection constraints between visual feature points and LiDAR 3-D points are used to compute the scaled camera motion,which is further utilized to align the drifted LiDAR scan with the camera image.Finally,the proposed method is sufficiently compared with two representative approaches in the online experiments with varying levels of random drift,then the method is further extended to the offline calibration experiment and is demonstrated by a comparison with two existing benchmark methods.

Constructing a Virtual Large Reference Plate with High-precision for Calibrating Cameras with Large FOV

It is well known that the accuracy of camera calibration is constrained by the size of the reference plate,it is difficult to fabricate large reference plates with high precision.Therefore,it is non-trivial to calibrate a camera with large field of view(FOV).In this paper,a method is proposed to construct a virtual large reference plate with high precision.Firstly,a high precision datum plane is constructed with a laser interferometer and one-dimensional air guideway,and then the reference plate is positioned at different locations and orientations in the FOV of the camera.The feature points of reference plate are projected to the datum plane to obtain a virtual large reference plate with high-precision.The camera is moved to several positions to get different virtual reference plates,and the camera is calibrated with the virtual reference plates.The experimental results show that the mean re-projection error of the camera calibrated with the proposed method is 0.062 pixels.The length of a scale bar with standard length of 959.778mm was measured with a vision system composed of two calibrated cameras,and the length measurement error is 0.389mm.

面向RGB-D语义分割的多模态任意旋转自监督学习

基于RGB-D数据的自监督学习受到广泛关注,然而大多数方法侧重全局级别的表示学习,会丢失对识别对象至关重要的局部细节信息.由于RGB-D数据中图像和深度具有几何一致性,因此这可以作为线索来指导RGB-D数据的自监督特征表示学习.在本文中,我们提出了ArbRot,它可以无限制地旋转角度并为代理任务生成多个伪标签用于自监督学习,而且还建立了全局和局部之间的上下文联系.本文所提出的ArbRot可以与其他对比学习方法联合训练,构建多模态多代理任务自监督学习框架,以增强图像和深度视图的特征表示一致性,从而为RGB-D语义分割任务提供有效的初始化.在SUN RGB-D和NYU Depth Dataset V2数据集上的实验结果表明,多模态任意旋转自监督学习得到的特征表示质量均高于基线模型.开源代码:https://github.com/Physu/ArbRot.

Analyzing the Impact of Scene Transitions on Indoor Camera Localization through Scene Change Detection in Real-Time

Real-time indoor camera localization is a significant problem in indoor robot navigation and surveillance systems.The scene can change during the image sequence and plays a vital role in the localization performance of robotic applications in terms of accuracy and speed.This research proposed a real-time indoor camera localization system based on a recurrent neural network that detects scene change during the image sequence.An annotated image dataset trains the proposed system and predicts the camera pose in real-time.The system mainly improved the localization performance of indoor cameras by more accurately predicting the camera pose.It also recognizes the scene changes during the sequence and evaluates the effects of these changes.This system achieved high accuracy and real-time performance.The scene change detection process was performed using visual rhythm and the proposed recurrent deep architecture,which performed camera pose prediction and scene change impact evaluation.Overall,this study proposed a novel real-time localization system for indoor cameras that detects scene changes and shows how they affect localization performance.