Optimized CUDA Implementation to Improve the Performance of Bundle Adjustment Algorithm on GPUs

The 3D reconstruction pipeline uses the Bundle Adjustment algorithm to refine the camera and point parameters. The Bundle Adjustment algorithm is a compute-intensive algorithm, and many researchers have improved its performance by implementing the algorithm on GPUs. In the previous research work, “Improving Accuracy and Computational Burden of Bundle Adjustment Algorithm using GPUs,” the authors demonstrated first the Bundle Adjustment algorithmic performance improvement by reducing the mean square error using an additional radial distorting parameter and explicitly computed analytical derivatives and reducing the computational burden of the Bundle Adjustment algorithm using GPUs. The naïve implementation of the CUDA code, a speedup of 10× for the largest dataset of 13,678 cameras, 4,455,747 points, and 28,975,571 projections was achieved. In this paper, we present the optimization of the Bundle Adjustment algorithm CUDA code on GPUs to achieve higher speedup. We propose a new data memory layout for the parameters in the Bundle Adjustment algorithm, resulting in contiguous memory access. We demonstrate that it improves the memory throughput on the GPUs, thereby improving the overall performance. We also demonstrate an increase in the computational throughput of the algorithm by optimizing the CUDA kernels to utilize the GPU resources effectively. A comparative performance study of explicitly computing an algorithm parameter versus using the Jacobians instead is presented. In the previous work, the Bundle Adjustment algorithm failed to converge for certain datasets due to several block matrices of the cameras in the augmented normal equation, resulting in rank-deficient matrices. In this work, we identify the cameras that cause rank-deficient matrices and preprocess the datasets to ensure the convergence of the BA algorithm. Our optimized CUDA implementation achieves convergence of the Bundle Adjustment algorithm in around 22 seconds for the largest dataset compared to 654 seconds for the sequential implementation, resulting in a speedup of 30×. Our optimized CUDA implementation presented in this paper has achieved a 3× speedup for the largest dataset compared to the previous naïve CUDA implementation.

A Self-calibration Bundle Adjustment Algorithm Based on Block Matrix Cholesky Decomposition Technology

In this study,the problem of bundle adjustment was revisited,and a novel algorithm based on block matrix Cholesky decomposition was proposed to solve the thorny problem of self-calibration bundle adjustment.The innovation points are reflected in the following aspects:①The proposed algorithm is not dependent on the Schur complement,and the calculation process is simple and clear;②The complexities of time and space tend to O(n)in the context of world point number is far greater than that of images and cameras,so the calculation magnitude and memory consumption can be reduced significantly;③The proposed algorithm can carry out self-calibration bundle adjustment in single-camera,multi-camera,and variable-camera modes;④Some measures are employed to improve the optimization effects.Experimental tests showed that the proposed algorithm has the ability to achieve state-of-the-art performance in accuracy and robustness,and it has a strong adaptability as well,because the optimized results are accurate and robust even if the initial values have large deviations from the truth.This study could provide theoretical guidance and technical support for the image-based positioning and 3D reconstruction in the fields of photogrammetry,computer vision and robotics.

Improving Accuracy and Computational Burden of Bundle Adjustment Algorithm Using GPUs

Bundle adjustment is a camera and point refinement technique in a 3D scene reconstruction pipeline. The camera parameters and the 3D points are refined by minimizing the difference between computed projection and observed projection of the image points formulated as a non-linear least-square problem. Levenberg-Marquardt method is used to solve the non-linear least-square problem. Solving the non-linear least-square problem is computationally expensive, proportional to the number of cameras, points, and projections. In this paper, we implement the Bundle Adjustment (BA) algorithm and analyze techniques to improve algorithmic performance by reducing the mean square error. We investigate using an additional radial distortion camera parameter in the BA algorithm and demonstrate better convergence of the mean square error. We also demonstrate the use of explicitly computed analytical derivatives. In addition, we implement the BA algorithm on GPUs using the CUDA parallel programming model to reduce the computational time burden of the BA algorithm. CUDA Streams, atomic operations, and cuBLAS library in the CUDA programming model are proposed, implemented, and demonstrated to improve the performance of the BA algorithm. Our implementation has demonstrated better convergence of the BA algorithm and achieved a speedup of up to 16× on the use of the BA algorithm on various datasets.

Vision based terrain reconstruction for planet rover using a special binocular bundle adjustment

This paper presents a pure vision based technique for 3D reconstruction of planet terrain. The reconstruction accuracy depends ultimately on an optimization technique known as 'bundle adjustment'. In vision techniques, the translation is only known up to a scale factor, and a single scale factor is assumed for the whole sequence of images if only one camera is used. If an extra camera is available, stereo vision based reconstruction can be obtained by binocular views. If the baseline of the stereo setup is known, the scale factor problem is solved. We found that direct application of classical bundle adjustment on the constraints inherent between the binocular views has not been tested. Our method incorporated this constraint into the conventional bundle adjustment method. This special binocular bundle adjustment has been performed on image sequences similar to planet terrain circumstances. Experimental results show that our special method enhances not only the localization accuracy, but also the terrain mapping quality.

BUNDLE ADJUSTMENTS CCD CAMERA CALIBRATION BASED ON COLLI NEARITY EQUATION

The solid template CCD camera calibration method of bundle adjustments basedon collinearity equation is presented considering the characteristics of space large-dimensionon-line measurement. In the method, a more comprehensive camera model is adopted which is based onthe pinhole model extended with distortions corrections. In the process of calibration, calibrationprecision is improved by imaging at different locations in the whole measurement space,multi-imaging at the same location and bundle adjustments optimization. The calibration experimentproves that the calibration method is able to fulfill calibration requirement of CCD camera appliedto vision measurement.

Accuracy Analysis on Bundle Adjustment of Remote Sensing Images Based on Dual Quaternion

A bundle adjustment method of remote sensing images based on dual quaternion is presented,which conducted the uniform disposal corresponding location and attitude of sequence images by the dual quaternion.The constraint relationship of image itself and sequence images is constructed to compensate the systematic errors.The feasibility of this method used in bundle adjustment is theoretically tested by the analysis of the structural characteristics of error equation and normal equation based on dual quaternion.Different distributions of control points and stepwise regression analysis are introduced into the experiment for RC30 image.The results show that the adjustment accuracy can achieve 0.2min plane and 1min elevation.As a result,this method provides a new technique for geometric location problem of remote sensing images.

Bundle adjustment for data processing of theodolite industrial surveying system

The photogrammetric bundle adjustment was used in data processing of electronic theodolite industrial surveying system by converting angular observations into virtual photo coordinates. The developed algorithm has ability of precision estimation and data snooping, do not need initial values of exterior orientation elements and object point coordinates. The form of control condition for the system is quite flexible. Neither centering nor leveling is the theodolite needed and the lay out of theodolite position is flexible when the system is used for precise survey. Experiments carried out in test field verify the validity of the data processing method. [

Distributed sparse bundle adjustment algorithm based on three-dimensional point partition and asynchronous communication

Sparse bundle adjustment(SBA) is a key but time-and memory-consuming step in three-dimensional(3 D) reconstruction. In this paper, we propose a 3 D point-based distributed SBA algorithm(DSBA) to improve the speed and scalability of SBA. The algorithm uses an asynchronously distributed sparse bundle adjustment(A-DSBA)to overlap data communication with equation computation. Compared with the synchronous DSBA mechanism(SDSBA), A-DSBA reduces the running time by 46%. The experimental results on several 3 D reconstruction datasets reveal that our distributed algorithm running on eight nodes is up to five times faster than that of the stand-alone parallel SBA. Furthermore, the speedup of the proposed algorithm(running on eight nodes with 48 cores) is up to41 times that of the serial SBA(running on a single node).

Hybrid Parallel Bundle Adjustment for 3D Scene Reconstruction with Massive Points

Bundle adjustment （BA） is a crucial but time consuming step in 3D reconstruction. In this paper, we intend to tackle a special class of BA problems where the reconstructed 3D points are much more numerous than the camera parameters, called Massive-Points BA （MPBA） problems. This is often the case when high-resolution images are used. We present a design and implementation of a new bundle adjustment algorithm for efficiently solving the MPBA problems. The use of hardware parallelism, the multi-core CPUs as well as GPUs, is explored. By careful memory-usage design, the graphic-memory limitation is effectively alleviated. Several modern acceleration strategies for bundle adjustment, such as the mixed-precision arithmetics, the embedded point iteration, and the preconditioned conjugate gradients, are explored and compared. By using several high-resolution image datasets, we generate a variety of MFBA problems, with which the performance of five bundle adjustment algorithms are evaluated. The experimental results show that our algorithm is up to 40 times faster than classical Sparse Bundle Adjustment, while maintaining comparable precision.

A low-overhead asynchronous consensus framework for distributed bundle adjustment

Generally,the distributed bundle adjustment(DBA)method uses multiple worker nodes to solve the bundle adjustment problems and overcomes the computation and memory storage limitations of a single computer.However,the performance considerably degrades owing to the overhead introduced by the additional block partitioning step and synchronous waiting.Therefore,we propose a low-overhead consensus framework.A partial barrier based asynchronous method is proposed to early achieve consensus with respect to the faster worker nodes to avoid waiting for the slower ones.A scene summarization procedure is designed and integrated into the block partitioning step to ensure that clustering can be performed on the small summarized scene.Experiments conducted on public datasets show that our method can improve the worker node utilization rate and reduce the block partitioning time.Also,sample applications are demonstrated using our large-scale culture heritage datasets.

Impact of a Bundle on Prevention and Control of Healthcare associated Infections in Intensive Care Unit

Inpatients in the intensive care unit（ICU） are at high risk for healthcare-associated infections（HAIs）. In the current study, a bundle of interventions and measures for preventing and controlling HAIs were developed and implemented in the ICU by trained personnel, and the impact of the bundle was evaluated. The incidence of HAIs, the adjusted daily incidence of HAIs and the incidence of three types of catheter-related infections before and after the bundle implementation were compared. The execution rate of the bundle for preventing and controlling ventilator-associated pneumonia（VAP） was increased from 82.06% in 2012 to 96.88% in 2013. The execution rate was increased from 83.03% in 2012 to 91.33% in 2013 for central line-associated bloodstream infection（CLABSI）, from 87.00% to 94.40% for catheter-associated urinary tract infection（CAUTI）, and from 82.05% to 98.55% for multidrug-resistant organisms（MDROs）, respectively. In total, 136 cases（10.37%） in 2012 and 113 cases（7.72%） in 2013 involved HAIs, respectively. Patients suffered from infection of the lower respiratory tract, the most common site of HAIs, in 134 cases（79.29%） in 2012 and 107 cases（74.30%） in 2013 respectively. The incidence of VAP was 32.72‰ and 24.60‰, the number of strains of pathogens isolated was 198 and 173, and the number of MDROs detected in the ICU was 91 and 74 in 2012 and 2013, respectively. The percentage of MDROs among the pathogens causing HAIs was decreased in each quarter of 2013 as compared with the corresponding percentage in 2012. In 2013, the execution rate of the bundle for preventing and controlling HAIs was increased, whereas the incidence of HAIs and VAP decreased as compared with that in 2012.

PRINCIPLE,SOFTWARE AND EXPERIMENT OF GPS-SUPPORTED AEROTRIANGULATION

In conventional aerial photogrammetry, the high accurate photogrammetric point determination is always carried out by aerotriangulation using a great deal of ground control points around the perimeter and in the center of block area because the exterior orien- tation parameters of aerial photographs are unknown. A technological revolution in pho- togrammetry has taken place since Navstar global positioning system (GPS) was applied to determine the 3D coordinates of exposure station positions during the photo flight missions. GPS-supported aerotriangulation is conducted by a combined bundle adjustment for pho- togrammetric observations and the camera orientation data. In this case, the essential ground control points are replaced by GPS-determined camera positions. Recent investigations show this method is coming to the practice. We have been engaged in the theoretical studies, soft- ware development, and related experiments and production in the field since 1990. So far the abundant research achievements are obtained in terms of the theory and application. In this paper,we first derives the mathematical model of GPS-supported aerotriangulation from the geometry between camera and airborne GPS antenna, then describes briefly a software pack- age WuCAPS (Wuhan combined adjustment program system) developed newly by the au- thor,which serves the purpose of the combined bundle adjustment for photogrammetric and non-photogrammetric observations. At the end of the present work, a set of actual aerial pho- tographs,at the image scale of 1: 34 000, with airborne GPS data taken from Tianjing site, China were processed by WuCAPS. The empirical results have verified that the accuracy of the combined bundle adjustment with 4 XYZ ground control points around the corners of block area is very close to that of the conventional bundle adjustment with 3 additional pa- rameters, that leads to reduce 88% field survey and 75% production cost, and can meet the specification of topographic mapping at small or medium scale by GPS-supported aerotriangu- lation without ground control. This shows the ample applicability and the economic benefit of kinematic GPS relative positioning in high accurate photogrammetric point determination.

Non-Metric CCD Camera Calibration Algorithm in a Digital Photogrammetry System

Camera calibration is a critical process in photogrammetry and a necessary step to acquire 3D information from a 2D image. In this paper, a flexible approach for CCD camera calibration using 2D direct linear transformation （DLT） and bundle adjustment is proposed. The proposed approach assumes that the camera interior orientation elements are known, and addresses a new closed form solution in planar object space based on homogenous coordinate representation and matrix factorization. Homogeneous coordinate representation offers a direct matrix correspondence between the parameters of the 2D DLT and the collinearity equation. The matrix factorization starts by recovering the elements of the rotation matrix and then solving for the camera position with the collinearity equation. Camera calibration with high precision is addressed by bundle adjustment using the initial values of the camera orientation elements. The results show that the calibration precision of principal point and focal length is about 0.2 and 0.3 pixels respectivelv, which can meet the requirements of close-range photogrammetry with high accuracy.

3D motion and geometric information system of single-antenna radar based on incomplete 1D range data

A 3D motion and geometric information system of single-antenna radar is proposed,which can be supported by spotlight synthetic aperture radar（SAR） system and inverse SAR（ISAR） system involving relative 3D motion of the rigid target.In this system,applying the geometry invariance of the rigid target,the unknown 3D shape and motion of the radar target can be reconstructed from the 1D range data of some scatterers extracted from the high-resolution range image.Compared with the current 1D-to-3D algorithm,in the proposed algorithm,the requirement of the 1D range data is expanded to incomplete formation involving large angular motion of the target and hence,the quantity of the scatterers and the abundance of 3D motion are enriched.Furthermore,with the three selected affine coordinates fixed,the multi-solution problem of the reconstruction is solved and the technique of nonlinear optimization can be successfully utilized in the system.Two simulations are implemented which verify the higher robustness of the system and the better performance of the 3D reconstruction for the radar target with unknown relative motion.

Positioning of Screw Holes Group Based on Digital Camera and Digital Control Drilling

Positioning of screw holes is an important production procedure for steel construction connecting with bolts. In this paper, a new production method is presented, in which the digital camera is used for taking pictures of screw holes and other techniques are advanced. This paper also indicates that the pixels of CCD chip in photogrammetry should be chosen as all geometric units in an image, such as interior elements and all kinds of distortions. The measure can also simplify the camera calibration for determining the size of non-square pixel.

System Structure and Calibration Models of Intelligent Photogrammetron

This paper describes the structure, geometric model and geometric calibrationof Photogrammetron Ⅰ - the first type of photogrammetron which is designed to be a coherent stereophotogrammetric system in which two cameras are mounted on a physical base but driven by anintelligent agent architecture. The system calibration is divided into two parts: the in-labcalibration determines the fixed parameters in advance of system operation, and the in-situcalibration keeps tracking the free parameters in real-time during the system operation. In a videosurveillance set-up, prepared control points are tracked in stereo image sequences, so that the freeparameters of the system can be continuously updated through iterative bundle adjustment and Kalmanfiltering.

Flexible Planar-Scene Camera Calibration Technique

A flexible camera calibration technique using 2D-DLT and bundle adjustment with planar scenes is proposed. The equation of principal line under image coordinate system represented with 2D-DLT parameters is educed using the correspondence between collinearity equations and 2D-DLT. A novel algorithm to obtain the initial value of principal point is put forward. Proof of Critical Motion Sequences for calibration is given in detail. The practical decomposition algorithm of exterior parameters using initial values of principal point, focal length and 2D-DLT parameters is discussed elaborately. Planar\|scene camera calibration algorithm with bundle adjustment is addressed. Very good results have been obtained with both computer simulations and real data calibration. The calibration result can be used in some high precision applications, such as reverse engineering and industrial inspection.

A preliminary study on photogrammetry for the FAST main reflector measurement

The Five-hundred-meter Aperture Spherical radio Telescope(FAST) is the largest single-dish radio telescope in the world. In this paper, we apply photogrammetry to measure local area surface accuracy of the FAST main reflector. Contrast with the existing photogrammetric methods that need to stick the cooperation target on the panel. In this paper, we directly detect nodes according to their natural feature. Analyzing the FAST reflector composition, we propose a two-step Hough transform method for node detection. Due to the high similarity of the neighboring area around nodes, it is hard to match two images by the feature matching method. Therefore, we apply the nodes combination approach to obtain the homography matrix between two photos for nodes matching. After nodes detection and matching,triangulation and bundle adjustment algorithms are adopted for 3 D reconstruction. During the experiment,the adjusted node position deform a local area of the FAST main reflector into a spherical cap with a radius of 300 m. The experimental result shows that the measurement accuracy of the sphere with a radius of 300 m is 12.299 mm, indicating that it is feasible to apply photogrammetry for the FAST main reflector measurement.

3-D Free-form Shape Measuring System Using Unconstrained Range Sensor

Three-dimensional（3-D） free-form shape measurement,a challenging task pursued by computer vision,is mainly characterized with single view acquisition and multiple view registration.Most of the conventional scanning systems are less flexibility and difficult to realize engineering applications for employing sequential registration tactic.To develop portable scanning system and engineering registration method overcoming problems of error accumulation and propagation is the research direction.In this paper,one 3-D free-form shape measuring system using unconstrained range sensor is designed.A quasi-active stereo binocular visual sensor embedded within a scanning mechanism is used as the range sensor.Error compensation is performed by residual amendment according to camera calibration lattice.Artificial control points are designed and adhered on object and one camera is introduced to shot these control points from different positions and orientations.Then ray bundle adjustment（BA） method is used to calculate the space coordinates of all the control points,so as to set up a global control net work.Registration can be completed by mapping at least 3 control points observed by range sensor in single view acquisition into the global control network.In this system,no calibration for laser plane is required and the motion of range sensor is completely free.The overlapping of neighboring region is unessential for registration.Therefore,the working range of the system can be easily extended.The measuring precision mainly depends on the quality of global control network.The sequential distances of coding control points are observed by electronic theodolites and then compared with those obtained according to BA result.Experimental results show that relative distance error of control points is no more than 0.2%.The proposed measuring system is portable,provides good capacity for global error control,and contributes to the engineering application of 3-D free-form shape measurement.

KLT-VIO:Real-time Monocular Visual-Inertial Odometry

This paper proposes a Visual-Inertial Odometry(VIO)algorithm that relies solely on monocular cameras and Inertial Measurement Units(IMU),capable of real-time self-position estimation for robots during movement.By integrating the optical flow method,the algorithm tracks both point and line features in images simultaneously,significantly reducing computational complexity and the matching time for line feature descriptors.Additionally,this paper advances the triangulation method for line features,using depth information from line segment endpoints to determine their Plcker coordinates in three-dimensional space.Tests on the EuRoC datasets show that the proposed algorithm outperforms PL-VIO in terms of processing speed per frame,with an approximate 5%to 10%improvement in both relative pose error(RPE)and absolute trajectory error(ATE).These results demonstrate that the proposed VIO algorithm is an efficient solution suitable for low-computing platforms requiring real-time localization and navigation.