Enhancing Dense Small Object Detection in UAV Images Based on Hybrid Transformer

Transformer-based models have facilitated significant advances in object detection.However,their extensive computational consumption and suboptimal detection of dense small objects curtail their applicability in unmanned aerial vehicle(UAV)imagery.Addressing these limitations,we propose a hybrid transformer-based detector,H-DETR,and enhance it for dense small objects,leading to an accurate and efficient model.Firstly,we introduce a hybrid transformer encoder,which integrates a convolutional neural network-based cross-scale fusion module with the original encoder to handle multi-scale feature sequences more efficiently.Furthermore,we propose two novel strategies to enhance detection performance without incurring additional inference computation.Query filter is designed to cope with the dense clustering inherent in drone-captured images by counteracting similar queries with a training-aware non-maximum suppression.Adversarial denoising learning is a novel enhancement method inspired by adversarial learning,which improves the detection of numerous small targets by counteracting the effects of artificial spatial and semantic noise.Extensive experiments on the VisDrone and UAVDT datasets substantiate the effectiveness of our approach,achieving a significant improvement in accuracy with a reduction in computational complexity.Our method achieves 31.9%and 21.1%AP on the VisDrone and UAVDT datasets,respectively,and has a faster inference speed,making it a competitive model in UAV image object detection.

A Novel Unsupervised Change Detection Method with Structure Consistency and GFLICM Based on UAV Images

With the rapid development of Unmanned Aerial Vehicle(UAV)technology,change detection methods based on UAV images have been extensively studied.However,the imaging of UAV sensors is susceptible to environmental interference,which leads to great differences of same object between UAV images.Overcoming the discrepancy difference between UAV images is crucial to improving the accuracy of change detection.To address this issue,a novel unsupervised change detection method based on structural consistency and the Generalized Fuzzy Local Information C-means Clustering Model(GFLICM)was proposed in this study.Within this method,the establishment of a graph-based structural consistency measure allowed for the detection of change information by comparing structure similarity between UAV images.The local variation coefficient was introduced and a new fuzzy factor was reconstructed,after which the GFLICM algorithm was used to analyze difference images.Finally,change detection results were analyzed qualitatively and quantitatively.To measure the feasibility and robustness of the proposed method,experiments were conducted using two data sets from the cities of Yangzhou and Nanjing.The experimental results show that the proposed method can improve the overall accuracy of change detection and reduce the false alarm rate when compared with other state-of-the-art change detection methods.

Pine wilt disease detection in high-resolution UAV images using object-oriented classification

Pine wilt disease(PWD)is currently one of the main causes of large-scale forest destruction.To control the spread of PWD,it is essential to detect affected pine trees quickly.This study investigated the feasibility of using the object-oriented multi-scale segmentation algorithm to identify trees discolored by PWD.We used an unmanned aerial vehicle(UAV)platform equipped with an RGB digital camera to obtain high spatial resolution images,and multiscale segmentation was applied to delineate the tree crown,coupling the use of object-oriented classification to classify trees discolored by PWD.Then,the optimal segmentation scale was implemented using the estimation of scale parameter(ESP2)plug-in.The feature space of the segmentation results was optimized,and appropriate features were selected for classification.The results showed that the optimal scale,shape,and compactness values of the tree crown segmentation algorithm were 56,0.5,and 0.8,respectively.The producer’s accuracy(PA),user’s accuracy(UA),and F1 score were 0.722,0.605,and 0.658,respectively.There were no significant classification errors in the final classification results,and the low accuracy was attributed to the low number of objects count caused by incorrect segmentation.The multi-scale segmentation and object-oriented classification method could accurately identify trees discolored by PWD with a straightforward and rapid processing.This study provides a technical method for monitoring the occurrence of PWD and identifying the discolored trees of disease using UAV-based high-resolution images.

Individual tree segmentation and biomass estimation based on UAV Digital aerial photograph

Digital aerial photograph(DAP)data is processed based on Structure from Motion(Sf M)algorithm and regional net adjustment method to generate digital surface discrete point clouds similar to Light Detection and Ranging(LiDAR)and digital orthophoto mosaic(DOM)similar to optical remote sensing image.In this study,we obtained highresolution images of mature forests of Chinese fir by unmanned aerial vehicle(UAV)flying through crossroute flight,and then reconstructed the threedimensional point clouds in the UAV aerial area by SfM technique.The point cloud segmentation(PCS)algorithm was used for the individual tree segmentation,and the F-score of the three sample plots were 0.91,0.94,and 0.94,respectively.Individual tree biomass modeling was conducted using 155 mature Chinese fir forests which were correctly segmented.The relative root mean squared error(rRMSE)values of random forest(RF),bagged tree(BT)and support vector regression(SVR)were 34.48%,35.74%and 40.93%,respectively.Our study demonstrated that DAP point clouds had great potential to extract forest vertical parameters and could be applied successfully in individual tree segmentation and individual tree biomass modeling.

Analysis of large-scale UAV images using a multi-scale hierarchical representation

Unmanned aerial vehicle(UAV)-based imaging systems have many superiorities compared with other platforms,such as high flexibility and low cost in collecting images,providing wide application prospects.However,the acquisition of the UAV-based image commonly results in very high resolution and very large-scale images,which poses great challenges for subsequent applications.Therefore,an efficient representation of large-scale UAV images is necessary for the extraction of the required information in a reasonable time.In this work,we proposed a multi-scale hierarchical representation,i.e.binary partition tree,for analyzing large-scale UAV images.More precisely,we first obtained an initial partition of images by an oversegmentation algorithm,i.e.the simple linear iterative clustering.Next,we merged the similar superpixels to build an object-based hierarchical structure by fully considering the spectral and spatial information of the superpixels and their topological relationships.Moreover,objects of interest and optimal segmentation were obtained using object-based analysis methods with the hierarchical structure.Experimental results on processing the post-seismic UAV images of the 2013 Ya’an earthquake and the mosaic of images in the South-west of Munich demonstrate the effectiveness and efficiency of our proposed method.

Super-resolution enhancement of UAV images based on fractional calculus and POCS

A super-resolution enhancement algorithm was proposed based on the combination of fractional calculus and Projection onto Convex Sets(POCS)for unmanned aerial vehicles(UAVs)images.The representative problems of UAV images including motion blur,fisheye effect distortion,overexposed,and so on can be improved by the proposed algorithm.The fractional calculus operator is used to enhance the high-resolution and low-resolution reference frames for POCS.The affine transformation parameters between low-resolution images and reference frame are calculated by Scale Invariant Feature Transform(SIFT)for matching.The point spread function of POCS is simulated by a fractional integral filter instead of Gaussian filter for more clarity of texture and detail.The objective indices and subjective effect are compared between the proposed and other methods.The experimental results indicate that the proposed method outperforms other algorithms in most cases,especially in the structure and detail clarity of the reconstructed images.

Land cover classification in a mixed forest-grassland ecosystem using LResU-net and UAV imagery

Using an unmanned aerial vehicle (UAV) paired with image semantic segmentation to classify land cover within natural vegetation can promote the development of forest and grassland field. Semantic segmentation normally excels in medical and building classification, but its usefulness in mixed forest-grassland ecosystems in semi-arid to semi-humid climates is unknown. This study proposes a new semantic segmentation network of LResU-net in which residual convolution unit (RCU) and loop convolution unit (LCU) are added to the U-net framework to classify images of different land covers generated by UAV high resolution. The selected model enhanced classification accuracy by increasing gradient mapping via RCU and modifying the size of convolution layers via LCU as well as reducing convolution kernels. To achieve this objective, a group of orthophotos were taken at an altitude of 260 m for testing in a natural forest-grassland ecosystem of Keyouqianqi, Inner Mongolia, China, and compared the results with those of three other network models (U-net, ResU-net and LU-net). The results show that both the highest kappa coefficient (0.86) and the highest overall accuracy (93.7%) resulted from LResU-net, and the value of most land covers provided by the producer’s and user’s accuracy generated in LResU-net exceeded 0.85. The pixel-area ratio approach was used to calculate the real areas of 10 different land covers where grasslands were 67.3%. The analysis of the effect of RCU and LCU on the model training performance indicates that the time of each epoch was shortened from U-net (358 s) to LResU-net (282 s). In addition, in order to classify areas that are not distinguishable, unclassified areas were defined and their impact on classification. LResU-net generated significantly more accurate results than the other three models and was regarded as the most appropriate approach to classify land cover in mixed forest-grassland ecosystems.

Comparative Analysis of Fractional Vegetation Cover Estimation Based on Multi-sensor Data in a Semi-arid Sandy Area

The estimation of fractional vegetation cover(FVC) is important for identifying and monitoring desertification, especially in arid and semiarid regions. By using regression and pixel dichotomy models, we present the comparison of Sentinel-2A(S2) multispectral instrument(MSI) and Landsat 8(L8) operational land imager(OLI) data regarding the retrieval of FVC in a semi-arid sandy area(Mu Us Sandland, China, in August 2016). A combination of unmanned aerial vehicle(UAV) high-spatial-resolution images and field plots were used to produce verified data. Based on a normalized difference vegetation index(NDVI) regression model, the results showed that, compared with that of L8, the coefficient of determination(R2) of S2 increased by 26.0%, and the root mean square error(RMSE) and the sum of absolute error(SAE) decreased by 3.0% and 11.4%, respectively. For the ratio vegetation index(RVI) regression model, compared with that of L8, the R2 of S2 increased by 26.0%, and the RMSE and SAE decreased by 8.0% and 20.0%, respectively. When the pixel dichotomy model was used, compared with that of L8, the RMSE of S2 decreased by 21.3%, and the SAE decreased by 26.9%. Overall, S2 performed better than L8 in terms of FVC inversion. Additionally, in this paper, we develop a verified scheme based on UAV data in combination with the object-based classification method. This scheme is feasible and sufficiently robust for building relationships between field data and inversion results from satellite data. Further, the synergy of multi-source sensors(especially UAVs and satellites) is a potential effective way to estimate and evaluate regional ecological environmental parameters(FVC).

Measuring loblolly pine crowns with drone imagery through deep learning

In modeling forest stand growth and yield,crown width,a measure for stand density,is among the parameters that allows for estimating stand timber volumes.However,accurately measuring tree crown size in the field,in particular for mature trees,is challenging.This study demonstrated a novel method of applying machine learning algorithms to aerial imagery acquired by an unmanned aerial vehicle(UAV)to identify tree crowns and their widths in two loblolly pine plantations in eastern Texas,USA.An ortho mosaic image derived from UAV-captured aerial photos was acquired for each plantation(a young stand before canopy closure,a mature stand with a closed canopy).For each site,the images were split into two subsets:one for training and one for validation purposes.Three widely used object detection methods in deep learning,the Faster region-based convolutional neural network(Faster R-CNN),You Only Look Once version 3(YOLOv3),and single shot detection(SSD),were applied to the training data,respectively.Each was used to train the model for performing crown recognition and crown extraction.Each model output was evaluated using an independent test data set.All three models were successful in detecting tree crowns with an accuracy greater than 93%,except the Faster R-CNN model that failed on the mature site.On the young site,the SSD model performed the best for crown extraction with a coefficient of determination(R^(2))of 0.92,followed by Faster R-CNN(0.88)and YOLOv3(0.62).As to the mature site,the SSD model achieved a R^(2)as high as 0.94,follow by YOLOv3(0.69).These deep leaning algorithms,in particular the SSD model,proved to be successfully in identifying tree crowns and estimating crown widths with satisfactory accuracy.For the purpose of forest inventory on loblolly pine plantations,using UAV-captured imagery paired with the SSD object detention application is a cost-effective alternative to traditional ground measurement.

Adaptive cropping shallow attention network for defect detection of bridge girder steel using unmanned aerial vehicle images

Bridges are an important part of railway infrastructure and need regular inspection and maintenance.Using unmanned aerial vehicle(UAV)technology to inspect railway infrastructure is an active research issue.However,due to the large size of UAV images,flight distance,and height changes,the object scale changes dramatically.At the same time,the elements of interest in railway bridges,such as bolts and corrosion,are small and dense objects,and the sample data set is seriously unbalanced,posing great challenges to the accurate detection of defects.In this paper,an adaptive cropping shallow attention network(ACSANet)is proposed,which includes an adaptive cropping strategy for large UAV images and a shallow attention network for small object detection in limited samples.To enhance the accuracy and generalization of the model,the shallow attention network model integrates a coordinate attention(CA)mechanism module and an alpha intersection over union(α-IOU)loss function,and then carries out defect detection on the bolts,steel surfaces,and railings of railway bridges.The test results show that the ACSANet model outperforms the YOLOv5s model using adaptive cropping strategy in terms of the total mAP(an evaluation index)and missing bolt mAP by 5%and 30%,respectively.Also,compared with the YOLOv5s model that adopts the common cropping strategy,the total mAP and missing bolt mAP are improved by 10%and 60%,respectively.Compared with the YOLOv5s model without any cropping strategy,the total mAP and missing bolt mAP are improved by 40%and 67%,respectively.

The co-effect of image resolution and crown size on deep learning for individual tree detection and delineation

Individual tree detection and delineation(ITDD)is an important subject in forestry and urban forestry.This study represents the first research to propose the concept of crown resolution to comprehensively evaluate the co-effect of image resolution and crown size on deep learning.Six images with different resolutions were derived from a DJI Unmanned Aerial Vehicle(UAV),and 1344 manually delineated Chinese fir(Cunninghamia lanceolata(Lamb)Hook)tree crowns were used for six training and validation mask region-based convolutional neural network(Mask R-CNN)models,while additional 476 delineated tree crowns were reserved for testing.The overall detection accuracy,the influence of different crown sizes,and crown resolutions were calculated to evaluate model performance accuracy with different image resolutions for ITDD.Results show that the highest accuracy was achieved when the crown resolution was between 800 and 12800 pixels/tree.The accuracy of ITDD was impacted by crown resolution,and it was unable to effectively identify Chinese fir when the crown resolution was less than 25 pixels/tree or higher than 12800 pixels/tree.The study highlights crown resolution as a critical factor affecting ITDD and suggests selecting the appropriate resolution based on the target detected crown size.