FusionMLS:Highly dynamic 3D reconstruction with consumergrade RGB-D cameras

Multi-view dynamic three-dimensional reconstruction has typically required the use of custom shutter-synchronized camera rigs in order to capture scenes containing rapid movements or complex topology changes. In this paper, we demonstrate that multiple unsynchronized low-cost RGB-D cameras can be used for the same purpose. To alleviate issues caused by unsynchronized shutters, we propose a novel depth frame interpolation technique that allows synchronized data capture from highly dynamic 3 D scenes. To manage the resulting huge number of input depth images, we also introduce an efficient moving least squares-based volumetric reconstruction method that generates triangle meshes of the scene. Our approach does not store the reconstruction volume in memory,making it memory-efficient and scalable to large scenes.Our implementation is completely GPU based and works in real time. The results shown herein, obtained with real data, demonstrate the effectiveness of our proposed method and its advantages compared to stateof-the-art approaches.

High-quality indoor scene 3D reconstruction with RGB-D cameras:A brief review

High-quality 3D reconstruction is an important topic in computer graphics and computer vision with many applications,such as robotics and augmented reality.The advent of consumer RGB-D cameras has made a profound advance in indoor scene reconstruction.For the past few years,researchers have spent significant effort to develop algorithms to capture 3D models with RGB-D cameras.As depth images produced by consumer RGB-D cameras are noisy and incomplete when surfaces are shiny,bright,transparent,or far from the camera,obtaining highquality 3D scene models is still a challenge for existing systems.We here review high-quality 3D indoor scene reconstruction methods using consumer RGB-D cameras.In this paper,we make comparisons and analyses from the following aspects:(i)depth processing methods in 3D reconstruction are reviewed in terms of enhancement and completion,(ii)ICP-based,feature-based,and hybrid methods of camera pose estimation methods are reviewed,and(iii)surface reconstruction methods are reviewed in terms of surface fusion,optimization,and completion.The performance of state-of-the-art methods is also compared and analyzed.This survey will be useful for researchers who want to follow best practices in designing new high-quality 3D reconstruction methods.

Imaging simulation and analysis of attitude jitter effect on topographic mapping for lunar orbiter stereo optical cameras

The geometric accuracy of topographic mapping with high-resolution remote sensing images is inevita-bly affected by the orbiter attitude jitter.Therefore,it is necessary to conduct preliminary research on the stereo mapping camera equipped on lunar orbiter before launching.In this work,an imaging simulation method consid-ering the attitude jitter is presented.The impact analysis of different attitude jitter on terrain undulation is conduct-ed by simulating jitter at three attitude angles,respectively.The proposed simulation method is based on the rigor-ous sensor model,using the lunar digital elevation model(DEM)and orthoimage as reference data.The orbit and attitude of the lunar stereo mapping camera are simulated while considering the attitude jitter.Two-dimensional simulated stereo images are generated according to the position and attitude of the orbiter in a given orbit.Experi-mental analyses were conducted by the DEM with the simulated stereo image.The simulation imaging results demonstrate that the proposed method can ensure imaging efficiency without losing the accuracy of topographic mapping.The effect of attitude jitter on the stereo mapping accuracy of the simulated images was analyzed through a DEM comparison.

Infrastructure-Based Vehicle Localization System for Indoor Parking Lots Using RGB-D Cameras

Accurate vehicle localization is a key technology for autonomous driving tasks in indoor parking lots,such as automated valet parking.Additionally,infrastructure-based cooperative driving systems have become a means to realizing intelligent driving.In this paper,we propose a novel and practical vehicle localization system using infrastructure-based RGB-D cameras for indoor parking lots.In the proposed system,we design a depth data preprocessing method with both simplicity and efficiency to reduce the computational burden resulting from a large amount of data.Meanwhile,the hardware synchronization for all cameras in the sensor network is not implemented owing to the disadvantage that it is extremely cumbersome and would significantly reduce the scalability of our system in mass deployments.Hence,to address the problem of data distortion accompanying vehicle motion,we propose a vehicle localization method by performing template point cloud registration in distributed depth data.Finally,a complete hardware system was built to verify the feasibility of our solution in a real-world environment.Experiments in an indoor parking lot demonstrated the effectiveness and accuracy of the proposed vehicle localization system,with a maximum root mean squared error of 5 cm at 15Hz compared with the ground truth.

Neural Network Method for Colorimetry Calibration of Video Cameras

To transfer the color data from a device (video camera) dependent color space into a device? independent color space, a multilayer feedforward network with the error backpropagation (BP) learning rule, was regarded as a nonlinear transformer realizing the mapping from the RGB color space to CIELAB color space. A variety of mapping accuracy were obtained with different network structures. BP neural networks can provide a satisfactory mapping accuracy in the field of color space transformation for video cameras.

A Review of RGB-D Camera Calibration Methods

RGB-D camera is a new type of sensor,which can obtain the depth and texture information in an unknown 3D scene simultaneously,and they have been applied in various fields widely.In fact,when implementing such kinds of applications using RGB-D camera,it is necessary to calibrate it first.To the best of our knowledge,at present,there is no existing a systemic summary related to RGB-D camera calibration methods.Therefore,a systemic review of RGB-D camera calibration is concluded as follows.Firstly,the mechanism of obtained measurement and the related principle of RGB-D camera calibration methods are presented.Subsequently,as some specific applications need to fuse depth and color information,the calibration methods of relative pose between depth camera and RGB camera are introduced in Section 2.Then the depth correction models within RGB-D cameras are summarized and compared respectively in Section 3.Thirdly,considering that the angle of the view field of RGB-D camera is smaller and limited to some specific applications,we discuss the calibration models of relative pose among multiple RGB-D cameras in Section 4.At last,the direction and trend of RGB-D camera calibration are prospected and concluded.

6-DOF motion estimation using optical flow based on dual cameras

Because of its characteristics of simple algorithm and hardware, optical flow-based motion estimation has become a hot research field, especially in GPS-denied environment. Optical flow could be used to obtain the aircraft motion information, but the six-(degree of freedom)(6-DOF) motion still couldn't be accurately estimated by existing methods. The purpose of this work is to provide a motion estimation method based on optical flow from forward and down looking cameras, which doesn't rely on the assumption of level flight. First, the distribution and decoupling method of optical flow from forward camera are utilized to get attitude. Then, the resulted angular velocities are utilized to obtain the translational optical flow of the down camera, which can eliminate the influence of rotational motion on velocity estimation. Besides, the translational motion estimation equation is simplified by establishing the relation between the depths of feature points and the aircraft altitude. Finally, simulation results show that the method presented is accurate and robust.

PCA-based 3D Shape Reconstruction of Human Foot Using Multiple Viewpoint Cameras

This paper describes a multiple camera-based method to reconstruct the 3D shape of a human foot. From a foot database, an initial 3D model of the foot represented by a cloud of points is built. The shape parameters, which can characterize more than 92% of a foot, are defined by using the principal component analysis method. Then, using ＂active shape models＂, the initial 3D model is adapted to the real foot captured in multiple images by applying some constraints （edge points＇ distance and color variance）. We insist here on the experiment part where we demonstrate the efficiency of the proposed method on a plastic foot model, and also on real human feet with various shapes. We propose and compare different ways of texturing the foot which is needed for reconstruction. We present an experiment performed on the plastic foot model and on human feet and propose two different ways to improve the final 3D shapers accuracy according to the previous experiments＇ results. The first improvement proposed is the densification of the cloud of points used to represent the initial model and the foot database. The second improvement concerns the projected patterns used to texture the foot. We conclude by showing the obtained results for a human foot with the average computed shape error being only 1.06 mm.

Two cameras detect more lesions in the small-bowel than one

AIM:To explore the feasibility of dual camera capsule (DCC)small-bowel(SB)imaging and to examine if two cameras complement each other to detect more SB lesions.METHODS:Forty-one eligible,consecutive patients underwent DCC SB imaging.Two experienced investigators examined the videos and compared the total number of detected lesions to the number of lesions detected by each camera separately.Examination tolerability was assessed using a questionnaire.RESULTS:One patient was excluded.DCC cameras detected 68 positive findings(POS)in 20(50%)cases.Fifty of them were detected by the"yellow"camera,48 by the"green"and 28 by both cameras;44%(n=22)of the"yellow"camera’s POS were not detected by the"green"camera and 42%(n=20)of the"green" camera’s POS were not detected by the"yellow"camera.In two cases,only one camera detected significant findings.All participants had 216 findings of unknown significance(FUS).The"yellow","green"and both cameras detected 171,161,and 116 FUS,respectively;32%(n=55)of the"yellow"camera’s FUS were not detected by the"green"camera and 28%(n=45)of the"green"camera’s FUS were not detected by the "yellow"camera.There were no complications related to the examination,and 97.6%of the patients would repeat the examination,if necessary.CONCLUSION:DCC SB examination is feasible and well tolerated.The two cameras complement each other to detect more SB lesions.

Color compensation for multi-view video coding based on diversity of cameras

A novel color compensation method for multi-view video coding (MVC) is proposed, which efficiently exploits the inter-view dependencies between views with the existence of color mismatch caused by the diversity of cameras. A color compensation model is developed in RGB channels and then extended to YCbCr channels for practical use. A modified inter-view reference picture is constructed based on the color compensation model, which is more similar to the coding picture than the original inter-view reference picture. Moreover, the color compensation factors can be derived in both encoder and decoder, therefore no additional data need to be transmitted to the decoder. The experimental results show that the proposed method improves the coding efficiency of MVC and maintains good subjective quality.

Survey of 3D modeling using depth cameras

Three-dimensional(3D)modeling is an important topic in computer graphics and computer vision.In recent years,the introduction of consumer-grade depth cameras has resulted in profound advances in 3D modeling.Starting with the basic data structure,this survey reviews the latest developments of 3D modeling based on depth cameras,including research works on camera tracking,3D object and scene reconstruction,and high-quality texture reconstruction.We also discuss the future work and possible solutions for 3D modeling based on the depth camera.

基于轻量化目标检测网络的RGB-D视觉SLAM系统

RGB-D SLAM是一种利用深度相机实现同时定位和地图构建的技术。传统的视觉SLAM系统基于对静态环境的假设,然而实际环境中往往存在动态物体,这可能导致SLAM系统的位姿估计出现显著的偏差。针对这一问题,提出了基于轻量化的YOLOv8s目标检测的RGB-D视觉SLAM系统,采用Socket通信方式,将目标检测结果传给SLAM,然后利用Depth Value-RANSAC几何算法剔除检测框内的动态特征点,提高了SLAM系统在动态环境中的定位精度。实验使用TUM数据集进行验证,结果表明,本文系统精度相比ORB-SLAM2有明显提高。与其他SLAM系统相比,本文系统在精度和实时性上有不同程度的改进。

Modeling and Rectification of Rolling Shutter Effect in CMOS Aerial Cameras

Due to the electronic rolling shutter, high-speed Complementary Metal-Oxide Semiconductor( CMOS) aerial cameras are generally subject to geometric distortions,which cannot be perfectly corrected by conventional vision-based algorithms. In this paper we propose a novel approach to address the problem of rolling shutter distortion in aerial imaging. A mathematical model is established by the coordinate transformation method. It can directly calculate the pixel distortion when an aerial camera is imaging at arbitrary gesture angles.Then all pixel distortions form a distortion map over the whole CMOS array and the map is exploited in the image rectification process incorporating reverse projection. The error analysis indicates that within the margin of measuring errors,the final calculation error of our model is less than 1/2 pixel. The experimental results show that our approach yields good rectification performance in a series of images with different distortions. We demonstrate that our method outperforms other vision-based algorithms in terms of the computational complexity,which makes it more suitable for aerial real-time imaging.

Adaptive Human Tracking Across Non-overlapping Cameras in Depression Angles

To track human across non-overlapping cameras in depression angles for applications such as multi-airplane visual human tracking and urban multi-camera surveillance,an adaptive human tracking method is proposed,focusing on both feature representation and human tracking mechanism.Feature representation describes individual by using both improved local appearance descriptors and statistical geometric parameters.The improved feature descriptors can be extracted quickly and make the human feature more discriminative.Adaptive human tracking mechanism is based on feature representation and it arranges the human image blobs in field of view into matrix.Primary appearance models are created to include the maximum inter-camera appearance information captured from different visual angles.The persons appeared in camera are first filtered by statistical geometric parameters.Then the one among the filtered persons who has the maximum matching scale with the primary models is determined to be the target person.Subsequently,the image blobs of the target person are used to update and generate new primary appearance models for the next camera,thus being robust to visual angle changes.Experimental results prove the excellence of the feature representation and show the good generalization capability of tracking mechanism as well as its robustness to condition variables.

基于RGB-D视觉信息融合的带式输送机煤流量检测方法研究

在煤矿中,精确测量带式输送机煤流量对于煤矿生产和管理至关重要。为了进一步提高通过机器视觉方式检测带式输送机煤流量的精度,提出一种基于RGB-D视觉信息融合的煤流量检测方法,使用RGB-D相机采集煤流信息后,利用RGB图像对深度图像从不同尺度进行增强,运用K-means聚类算法,结合RGB图像对预分割的煤料区域深度图像进一步分割,基于微元法建立煤堆体积计算模型计算煤流量。实验结果表明,该方法的平均检测误差为1.57%,平均耗时263.77 ms,满足实际生产要求,为带式输送机煤流量检测提出了一种新的有效途径。

Optimized Method for Real-time Texture Reconstruction with RGB-D Camera

With the appearance of RGB-D camera, the field of three-dimensional (3D) reconstruction receives more and more attention. In this paper, we present an optimization approach to produce high-quality textured 3D models based on the real-time 3D reconstruction system. The resulting models of real-time texture reconstruction often suffer from blurring, ghosting, and other artifacts. Our approach addresses this texture quality problem using blur detection and an optimized weight function. Experimental results demonstrate that our approach can improve the quality of textured 3D models by reducing the blur and ghosts on the model surface. © 2017, Tianjin University and Springer-Verlag GmbH Germany.

Image Motion Compensation of Off-Axis TMA Three-Line ArrayAerospace Mapping Cameras

To enhance the image motion compensation accuracy of off-axis three-mirror anastigmatic( TMA)three-line array aerospace mapping cameras,a new method of image motion velocity field modeling is proposed in this paper. Firstly,based on the imaging principle of mapping cameras,an analytical expression of image motion velocity of off-axis TMA three-line array aerospace mapping cameras is deduced from different coordinate systems we established and the attitude dynamics principle. Then,the case of a three-line array mapping camera is studied,in which the simulation of the focal plane image motion velocity fields of the forward-view camera,the nadir-view camera and the backward-view camera are carried out,and the optimization schemes for image motion velocity matching and drift angle matching are formulated according the simulation results. Finally,this method is verified with a dynamic imaging experimental system. The results are indicative of that when image motion compensation for nadir-view camera is conducted using the proposed image motion velocity field model,the line pair of target images at Nyquist frequency is clear and distinguishable. Under the constraint that modulation transfer function( MTF) reduces by 5%,when the horizontal frequencies of the forward-view camera and the backward-view camera are adjusted uniformly according to the proposed image motion velocity matching scheme,the time delay integration( TDI) stages reach 6 at most. When the TDI stages are more than 6,the three groups of camera will independently undergo horizontal frequency adjustment. However, when the proposed drift angle matching scheme is adopted for uniform drift angle adjustment,the number of TDI stages will not exceed 81. The experimental results have demonstrated the validity and accuracy of the proposed image motion velocity field model and matching optimization scheme,providing reliable basis for on-orbit image motion compensation of aerospace mapping cameras.

Dual Dynamic PTZ Tracking Using Cooperating Cameras

This paper presents a real-time, dynamic system that uses high resolution gimbals and motorized lenses with position encoders on their zoom and focus elements to “recalibrate” the system as needed to track a target. Systems that initially calibrate for a mapping between pixels of a wide field of view (FOV) master camera and the pan-tilt (PT) settings of a steerable narrow FOV slave camera assume that the target is travelling on a plane. As the target travels through the FOV of the master camera, the slave cameras PT settings are then adjusted to keep the target centered within its FOV. In this paper, we describe a system we have developed that allows both cameras to move and extract the 3D coordinates of the target. This is done with only a single initial calibration between pairs of cameras and high-resolution pan-tilt-zoom (PTZ) platforms. Using the information from the PT settings of the PTZ platform as well as the precalibrated settings from a preset zoom lens, the 3D coordinates of the target are extracted and compared to those of a laser range finder and static-dynamic camera pair accuracies.

基于特征调节器和双路径引导的RGB-D室内语义分割

针对室内场景图像语义分割结果不精确、显著图粗糙的问题,提出一种基于多模态特征优化提取和双路径引导解码的网络架构(feature regulator and dual-path guidance,FG-Net)。具体来说,设计的特征调节器对每个阶段的多模态特征依次进行噪声过滤、重加权表示、差异性互补和交互融合,通过强化RGB和深度特征聚合,优化特征提取过程中的多模态特征表示。然后,在解码阶段引入特征交互融合后丰富的跨模态线索,进一步发挥多模态特征的优势。结合双路径协同引导结构,在解码阶段融合多尺度、多层次的特征信息,从而输出更细致的显著图。实验在公开数据集NYUD-v2和SUN RGB-D上进行,在主要评价指标mIoU上达到48.5%,优于其他先进算法。结果表明,该算法实现了更精细的室内场景图像语义分割,表现出了较好的泛化性和鲁棒性。