A Local Deviation Constraint Based Non-Rigid Structure From Motion Approach

In many traditional non-rigid structure from motion(NRSFM)approaches,the estimation results of part feature points may significantly deviate from their true values because only the overall estimation error is considered in their models.Aimed at solving this issue,a local deviation-constrained-based column-space-fitting approach is proposed in this paper to alleviate estimation deviation.In our work,an effective model is first constructed with two terms:the overall estimation error,which is computed by a linear subspace representation,and a constraint term,which is based on the variance of the reconstruction error for each frame.Furthermore,an augmented Lagrange multipliers(ALM)iterative algorithm is presented to optimize the proposed model.Moreover,a convergence analysis is performed with three steps for the optimization process.As both the overall estimation error and the local deviation are utilized,the proposed method can achieve a good estimation performance and a relatively uniform estimation error distribution for different feature points.Experimental results on several widely used synthetic sequences and real sequences demonstrate the effectiveness and feasibility of the proposed algorithm.

Applications of structure from motion: a survey

Structure from motion (SfM) has been an active research area in computer vision for decades and numerous practical applications are benefiting from this research. While no previous work has tried to summarize the applications appearing in the literature, this paper deals with a comprehensive overview of recent applications of SfM by classifying them into 10 categories, namely augmented reality, autonomous navigation/guidance, motion capture, hand-eye calibration, image/video processing, image-based 3D modeling, remote sensing, image organization/browsing, segmentation and recognition, and military applications. The goal is to provide insights for researchers to position their work more appropriately in the context of existing techniques, and to perceive both new applications and relevant research problems.

Quantifying spatial distribution of interrill and rill erosion in a loess at different slopes using structure from motion(SfM)photogrammetry

The spatial distribution of interrill and rill erosion is essential for unravelling soil erosion principles and the application of soil and water conservation practices.To quantify interrill and rill erosion and their spatial development,four 30-min rainfalls at 90 mm h^(-1)intensity were consecutively simulated on runoff plots packed with a loess at six slopes of 10°,15°,20°,25°,30°and 35°.The soil surface was measured using the structure from motion(SfM)photogrammetry upon each simulation run,and the runoff and sediment samples were collected and measured at every 10 min.Rills did not develop until the third simulation run.During the initial two runs,the lower third section was more severely eroded than the upper and middle thirds along the slope direction,yet the interrill erosion was statistically uniform from left to right.Rills tended to emerge by both sidewalls and in the lower portion in the third run.The corresponding rill erosion increased with slope from 10°to 20°and then decreased for the slopes steeper,which was consistent with the slope trend of the sediment yield directly measured.The rills expanded substantially primarily via head retreat and to a lesser extent via sideward erosion after receiving another 30-min rainfall.Rill erosion contributed 69.3%of the total erosion loss,and shifted the critical slope corresponding to the maximum loss from 20°to 25°.These findings demonstrate the significance of rill erosion not only in total soil loss but also in its relation to slope,as well as the effectiveness of SfM photogrammetry in quantifying interrill and rill erosion.

From landslide characterization to nature reserve management:The“Scialimata Grande di Torre Alfina”landslide Geosite(Central Apennines,Italy)

Italy is characterized by widespread geomorphological instability,among which landslides leave impressive marks on the landscape.Nevertheless,landslide bodies may represent key sites for thematic and educational itineraries,especially in protected areas,where their management becomes an important issue.Our study focuses on the"Monte Rufeno Nature Reserve"(Central Apennines,Italy),where iconic landslides are present.Here,the"Scialimata Grande di Torre Alfina"landslide(SGTA)is listed in the regional Geosite database.This work aims to propose a multiscale procedure for landslide analysis,in terms of both hazard sources but also educational and geoheritage enhancement opportunities in natural reserves.After performing a Landslide Susceptibility conditional Analysis(LSA)for the reserve territory,attention was focused on the SGTA,to define properly its features and morphodynamics.A multi-disciplinary approach was adopted,by applying both remote sensing(UAV structure from motion,Photointerpretation)and field survey(geomorphological and GPS monitoring).From the LSA,based on drainage density,curvature,and slope triggering factors,the road and trail susceptibility maps were derived,as base tools for future risk assessments and trail paths management within the reserve.At the SGTA scale,the monitoring showed a displacement of up to 23 m during the time interval between 2015 and 2018.The landslide dynamics seem to be driven by alternating dry and extremely wet periods;moreover,leaks from the aqueduct in the detachment area and piping effects through clays may have also decreased the substrate cohesion.The SGTA complex influence on the Paglia River valley geometry was also hypothesized,underlining the action of landslide through different spatial scales(on-site and off-site)and on different environment features(sediment connectivity,hydrology).Finally,the SGTA appears highly representative of the geomorphic dynamics within the Nature Reserve(i.e.,scientific value)and it could be classified as an active geosite.Since the site was featured by a tourist trail,adequate management strategies must be adopted,considering the educational value and safety issues.

Individual tree segmentation and biomass estimation based on UAV Digital aerial photograph

Digital aerial photograph(DAP)data is processed based on Structure from Motion(Sf M)algorithm and regional net adjustment method to generate digital surface discrete point clouds similar to Light Detection and Ranging(LiDAR)and digital orthophoto mosaic(DOM)similar to optical remote sensing image.In this study,we obtained highresolution images of mature forests of Chinese fir by unmanned aerial vehicle(UAV)flying through crossroute flight,and then reconstructed the threedimensional point clouds in the UAV aerial area by SfM technique.The point cloud segmentation(PCS)algorithm was used for the individual tree segmentation,and the F-score of the three sample plots were 0.91,0.94,and 0.94,respectively.Individual tree biomass modeling was conducted using 155 mature Chinese fir forests which were correctly segmented.The relative root mean squared error(rRMSE)values of random forest(RF),bagged tree(BT)and support vector regression(SVR)were 34.48%,35.74%and 40.93%,respectively.Our study demonstrated that DAP point clouds had great potential to extract forest vertical parameters and could be applied successfully in individual tree segmentation and individual tree biomass modeling.

基于无人机和SfM的天津港堆场散料体积测量

随着智慧港口的快速发展,露天堆场的自动化、智能化管控越来越重要,其中准确快速测量露天堆场散装物料体积、堆场占用率等物理量是堆场智能化管理发展的重要工作。以天津港典型堆场为例,应用大疆悟二飞行器、禅思X5S云台相机、Agisoft Metashape软件,基于无人机摄影测量与SfM技术,建立了堆场散装物料的数字高程模型,得到了堆场料堆分布图,计算了料堆总体积和堆场占用率,研究了模型参数设置对测量精度和效率造成的影响。结果表明:该方法可快速准确地获得堆场料堆总体积和堆场占用率,每平方千米堆场的测量内外业耗时1h左右,料堆体积测量相对误差小于3%,堆场占用率测量绝对误差小于0.5%,满足堆场测量的实际需求。模型参数对最终的计算结果精度没有显著影响,但是模型参数设置越高,生成模型耗费的时间就越长,因此模型参数设置为最低时,时间成本最低,在大范围堆场测量中具有实用意义。基于无人机和SfM快速测量堆场散料体积的方法可改变传统的大型堆场管理模式。研究成果可以为堆场散料体积快速测量与堆场智能管控提供技术支撑。

Deep-Learning-Empowered 3D Reconstruction for Dehazed Images in IoT-Enhanced Smart Cities

With increasingly more smart cameras deployed in infrastructure and commercial buildings,3D reconstruction can quickly obtain cities’information and improve the efficiency of government services.Images collected in outdoor hazy environments are prone to color distortion and low contrast;thus,the desired visual effect cannot be achieved and the difficulty of target detection is increased.Artificial intelligence(AI)solutions provide great help for dehazy images,which can automatically identify patterns or monitor the environment.Therefore,we propose a 3D reconstruction method of dehazed images for smart cities based on deep learning.First,we propose a fine transmission image deep convolutional regression network(FT-DCRN)dehazing algorithm that uses fine transmission image and atmospheric light value to compute dehazed image.The DCRN is used to obtain the coarse transmission image,which can not only expand the receptive field of the network but also retain the features to maintain the nonlinearity of the overall network.The fine transmission image is obtained by refining the coarse transmission image using a guided filter.The atmospheric light value is estimated according to the position and brightness of the pixels in the original hazy image.Second,we use the dehazed images generated by the FT-DCRN dehazing algorithm for 3D reconstruction.An advanced relaxed iterative fine matching based on the structure from motion(ARI-SFM)algorithm is proposed.The ARISFM algorithm,which obtains the fine matching corner pairs and reduces the number of iterations,establishes an accurate one-to-one matching corner relationship.The experimental results show that our FT-DCRN dehazing algorithm improves the accuracy compared to other representative algorithms.In addition,the ARI-SFM algorithm guarantees the precision and improves the efficiency.

A new approach to study terrestrial yardang geomorphology based on high-resolution data acquired by unmanned aerial vehicles(UAVs): A showcase of whaleback yardangs in Qaidam Basin, NW China

Yardangs are wind-eroded ridges usually observed in arid regions on Earth and other planets. Previous geomorphology studies of terrestrial yardang fields depended on satellite data and limited fieldwork. The geometry measurements of those yardangs based on satellite data are limited to the length, the width, and the spacing between the yardangs; elevations could not be studied due to the relatively low resolution of the satellite acquired elevation data, e.g. digital elevation models(DEMs). However, the elevation information(e.g. heights of the yardang surfaces) and related information(e.g. slope) of the yardangs are critical to understanding the characteristics and evolution of these aeolian features. Here we report a novel approach, using unmanned aerial vehicles(UAVs) to generate centimeterresolution orthomosaics and DEMs for the study of whaleback yardangs in Qaidam Basin, NW China. The ultra-high-resolution data provide new insights into the geomorphology characteristics and evolution of the whaleback yardangs in Qaidam Basin. These centimeter-resolution datasets also have important potential in:(1) high accuracy estimation of erosion volume;(2) modeling in very fine scale of wind dynamics related to yardang formation;(3) detailed comparative planetary geomorphology study for Mars, Venus, and Titan.

Using Pure Translation to Get Euclidean Reconstruction

A technique for getting Euclidean reconstruction from two images of the same scene taken by a single moving camera, which undergoes a pure translation, is presented. Euclidean reconstruction of the scene up to three scale factors can be obtained by using this special but still realistic motion when the skew factor of the cam- era is zero; otherwise Euclidean reconstruction of the depth up to one scale factor can be achieved. The only assumption is that the camera intrinsic parameters are constant. Using this special but still realistic motion to do the reconstruction has the advantage that no projective reconstruction is needed and the Euclidean reconstruction is computed directly from the point correspondences in the two images.

Accuracy of common stem volume formulae using terrestrial photogrammetric point clouds:a case study with savanna trees in Benin

Recent applications of digital photogrammetry in forestry have highlighted its utility as a viable mensuration technique.However,in tropical regions little research has been done on the accuracy of this approach for stem volume calculation.In this study,the performance of Structure from Motion photogrammetry for estimating individual tree stem volume in relation to traditional approaches was evaluated.We selected 30 trees from five savanna species growing at the periphery of the W National Park in northern Benin and measured their circumferences at different heights using traditional tape and clinometer.Stem volumes of sample trees were estimated from the measured circumferences using nine volumetric formulae for solids of revolution,including cylinder,cone,paraboloid,neiloid and their respective fustrums.Each tree was photographed and stem volume determined using a taper function derived from tri-dimensional stem models.This reference volume was compared with the results of formulaic estimations.Tree stem profiles were further decomposed into different portions,approximately corresponding to the stump,butt logs and logs,and the suitability of each solid of revolution was assessed for simulating the resulting shapes.Stem volumes calculated using the fustrums of paraboloid and neiloid formulae were the closest to reference volumes with a bias and root mean square error of 8.0%and 24.4%,respectively.Stems closely resembled fustrums of a paraboloid and a neiloid.Individual stem portions assumed different solids as follows:fustrums of paraboloid and neiloid were more prevalent from the stump to breast height,while a paraboloid closely matched stem shapes beyond this point.Therefore,a more accurate stem volumetric estimate was attained when stems were considered as a composite of at least three geometric solids.

GNSS约束的长航带无人机影像自检校方法

相机自检校直接决定无人机影像空三的精度。沿输电线路走廊采集的长航带结构无人机影像是一种典型的退化配置,对其自检校容易出现“碗状”效应。为解决该问题,传统方法往往依赖较多控制点,而提出的自检校方法仅需一个控制点。首先研究经典物理模型和最新的数学模型;然后在增量式SfM(structure from motion)框架下,设计了一种联合无人机影像相机检校参数初始化和高精度差分全球导航卫星系统(global navigation satellite system,GNSS)位置信息辅助的相机自检校方法。利用两个实验区域不同采集模式下的4组无人机电力走廊影像进行无控制约束以及单个控制点约束的相机自检校实验。结果表明,提出的相机自检校策略在无控制点约束时,可以有效缓解长航带结构空三的“碗状”效应,减轻模型的弯曲程度,提高自检校空三的绝对精度;单个控制点约束自检校时,水平和高程精度均优于0.06 m。与当前主流开源和商业软件对比,该算法能够得到相当或更高精度。

基于卷帘快门相机模型的高精度建图与视觉定位

使用卷帘快门相机进行移动测量时,由于逐行曝光的特点,影像上会产生果冻效应。而传统的建图与视觉定位方法大多假设影像是全局快门获取,直接处理卷帘快门影像很难得到高精度的结果。针对以上问题,构建基于插值的卷帘快门相机模型,并将其用于卷帘快门影像的高精度建图与视觉定位。建图时使用运动恢复结构算法,利用卷帘快门相机模型对影像特征点进行坐标插值,获取该点的位姿,继而得到对应的投影矩阵,再进行卷帘快门三角化和卷帘快门光束法平差优化,完成高精度建图。在视觉定位环节,利用PnP(perspective-n-point)算法获取影像定位初值,之后使用卷帘快门影像绝对位姿优化算法进行优化。实验结果表明,与传统方法相比,所提方法的建图与视觉定位精度更高。

Geo-registered 3D models from crowdsourced image collections

In this article we present our system for scalable,robust,and fast city-scale reconstruction from Internet photo collections(IPC)obtaining geo-registered dense 3D models.The major achievements of our system are the efficient use of coarse appearance descriptors combined with strong geometric constraints to reduce the computational complexity of the image overlap search.This unique combination of recognition and geometric constraints allows our method to reduce from quadratic complexity in the number of images to almost linear complexity in the IPC size.Accordingly,our 3D-modeling framework is inherently better scalable than other state of the art methods and in fact is currently the only method to support modeling from millions of images.In addition,we propose a novel mechanism to overcome the inherent scale ambiguity of the reconstructed models by exploiting geo-tags of the Internet photo collection images and readily available StreetView panoramas for fully automatic geo-registration of the 3D model.Moreover,our system also exploits image appearance clustering to tackle the challenge of computing dense 3D models from an image collection that has significant variation in illumination between images along with a wide variety of sensors and their associated different radiometric camera parameters.Our algorithm exploits the redundancy of the data to suppress estimation noise through a novel depth map fusion.The fusion simultaneously exploits surface and free space constraints during the fusion of a large number of depth maps.Cost volume compression during the fusion achieves lower memory requirements for high-resolution models.We demonstrate our system on a variety of scenes from an Internet photo collection of Berlin containing almost three million images from which we compute dense models in less than the span of a day on a single computer.

地面摄影RTK辅助SFM算法的影像定向

基于运动恢复结构(structure from motion,SFM)算法的地面影像定向通常没有考虑影像的位置姿态信息,影像定向结果在自由网坐标系,因此无法获取绝对坐标和真实尺度;利用地面摄影实时动态(real-time kinematic,RTK)技术可以获取地面影像的位置姿态信息,但是RTK和摄影测量的角度系统定义不同,RTK相位中心与影像投影中心存在偏差。针对这些问题,推导了利用RTK获取的GPS和角元素heading pitch roll(HPR)计算影像外方位元素的公式,提出地面摄影RTK辅助SFM算法的地面影像定向方法,在少量或者无需像控点条件下将SFM算法自由网的定向结果转换到绝对坐标系下,获得场景的真实坐标和尺度。实验结果表明,该转换公式正确,地面摄影RTK辅助SFM算法的影像定向方法可行。

Hybrid Parallel Bundle Adjustment for 3D Scene Reconstruction with Massive Points

Bundle adjustment （BA） is a crucial but time consuming step in 3D reconstruction. In this paper, we intend to tackle a special class of BA problems where the reconstructed 3D points are much more numerous than the camera parameters, called Massive-Points BA （MPBA） problems. This is often the case when high-resolution images are used. We present a design and implementation of a new bundle adjustment algorithm for efficiently solving the MPBA problems. The use of hardware parallelism, the multi-core CPUs as well as GPUs, is explored. By careful memory-usage design, the graphic-memory limitation is effectively alleviated. Several modern acceleration strategies for bundle adjustment, such as the mixed-precision arithmetics, the embedded point iteration, and the preconditioned conjugate gradients, are explored and compared. By using several high-resolution image datasets, we generate a variety of MFBA problems, with which the performance of five bundle adjustment algorithms are evaluated. The experimental results show that our algorithm is up to 40 times faster than classical Sparse Bundle Adjustment, while maintaining comparable precision.

Novel SfM-DLT method for metro tunnel 3D reconstruction and Visualization

Metro tunnels play a crucial role in urban transportation.However,with growing tunnel operation periods,defects,and large deformations appearing,these are influencing tunnel structural performance and threatening public safety.Three-dimensional(3D)tunnel reconstruction is an effective way to highlight tunnel conditions and provide a basis for engineering management and maintenance.However,the current methods of tunnel 3D reconstruction do not sufficiently combine the qualitative and quantitative characteristics of tunnel states.In this study,a novel method for metro tunnel 3D reconstruction based on structure from motion(SfM)and direct linear transformation(DLT)is proposed.The dimensionless 3D reconstruction point cloud acquired through the SfM method showcases the qualitative characteristics(such as leakage and pipelines)of the tunnel state.The close-range photogrammetry DLT method provides scale information missing from the SfM method and quantitative characteristics(such as profile deformation)of the tunnel state.The SfM-DLT method was tested in a Shanghai metro tunnel,and proved to be feasible and promising for future tunnel inspections.

3D reconstruction and automatic leakage defect quantification of metro tunnel based on SfM-Deep learning method

Various structural defects deteriorate tunnel operation status and threaten public safety.Current tunnel inspection methods face problems of low efficiency,high equipment expense,and difficult data management.Combining the deep learning model and the 3D reconstruction method based on structure from motion(SfM),this paper proposes a novel SfM-Deep learning method for tunnel inspection.The high-quality 3D tunnel model is constructed by using images taken every 1 m along the longitudinal direction.The instance segmentation of leakage in longitudinal images is realized using the mask region-based convolutional neural network deep learning model.The SfM-Deep learning method projects the texture of the images after defect recognition to the 3D model and realizes the visualization of leakage defects.By projecting the model to the design cylindrical surface and expanding it,the tunnel leakage area is quantified.Through its practical application in a Shanghai metro shield tunnel,the reliability of the proposed method was verified.The novel SfM-Deep learning method can help engineers efficiently carry out intelligent tunnel detection.

Visual localization for asteroid touchdown operation based on local image features

In an asteroid sample-return mission,accurate position estimation of the spacecraft relative to the asteroid is essential for landing at the target point.During the missions of Hayabusa and Hayabusa2,the main part of the visual position estimation procedure was performed by human operators on the Earth based on a sequence of asteroid images acquired and sent by the spacecraft.Although this approach is still adopted in critical space missions,there is an increasing demand for automated visual position estimation,so that the time and cost of human intervention may be reduced.In this paper,we propose a method for estimating the relative position of the spacecraft and asteroid during the descent phase for touchdown from an image sequence using state-of-the-art techniques of image processing,feature extraction,and structure from motion.We apply this method to real Ryugu images that were taken by Hayabusa2 from altitudes of 20 km-500 m.It is demonstrated that the method has practical relevance for altitudes within the range of 5-1 km.This result indicates that our method could improve the efficiency of the ground operation in the global mapping and navigation during the touchdown sequence,whereas full automation and autonomous on-board estimation are beyond the scope of this study.Furthermore,we discuss the challenges of developing a completely automatic position estimation framework.

A comparison of three surface roughness characterization techniques:photogrammetry,pin profiler,and smartphone-based LiDAR

Surface roughness plays an important role in microwave remote sensing.In the agricultural domain,surface roughness is crucial for soil moisture retrieval methods that use electromagnetic surface scattering or microwave radiative transfer models.Therefore,improved characterization of Soil Surface Roughness(SSR)is of considerable importance.In this study,three approaches,including a standard pin profiler,a LiDAR point cloud generated from an iPhone 12 Pro,and a Structure from Motion(SfM)photogrammetric point cloud,were applied over 24 surface profiles with different roughness variations to measure surface roughness.The objective of this study was to evaluate the capability of smartphone-based LiDAR technology to measure surface roughness parameters and compare the results of this technique with the more common approaches.Results showed that the iPhone LiDAR technology,when point cloud data is captured in a fine-resolution mode,has a significant correlation with SfM photogrammetry(R2=0.70)and a relatively close agreement with pin profiler(R2=0.60).However,this accuracy tends to be greater for random surfaces and rough profiles with row structure orientations.The results of this study confirm that smartphone-based LiDAR can be used as a cost-effective,fast,and time-efficient alternative tool for measuring surface roughness,especially for rough,wide,and inaccessible areas.

Democratizing photogrammetry:an accuracy perspective

Photogrammetry is experiencing an era of democratization mostly due to the popularity and availability of many commercial off-the-shelf devices,such as drones and smartphones.They are used as the most convenient and effective tools for high-resolution image acquisition for a wide range of applications in science,engineering,management,and cultural heritage.However,the quality,particularly the geometric accuracy,of the outcomes from such consumer sensors is still unclear.Furthermore,the expected quality under different control schemes has yet to be thoroughly investigated.This paper intends to answer those questions with a comprehensive comparative evaluation.Photogrammetry,in particular,structure from motion,has been used to reconstruct a 3D building model from smartphone and consumer drone images,as well as from professional drone images,all under various ground control schemes.Results from this study show that the positioning accuracy of smartphone images under direct geo-referencing is 165.4 cm,however,this could be improved to 43.3 cm and 14.5 cm when introducing aerial lidar data and total station surveys as ground control,respectively.Similar results are found for consumer drone images as well.For comparison,this study shows the use of the professional drone is able to achieve a positioning accuracy of 3.7 cm.Furthermore,we demonstrate that through the combined use of drone and smartphone images we are able to obtain full coverage of the entire target with a 2.3 cm positioning accuracy.Our study concludes that smartphone images can achieve an accuracy equivalent to consumer drone images and can be used as the primary data source for building facade data collection.