基于Stereo Camera-YOLOv5自然环境下百香果检测与定位模型

针对百香果采摘机器人在自然环境中作业时受复杂光线及遮挡影响,难以快速精确地检测及定位成熟百香果的问题,提出一种基于Stereo Camera-YOLOv5自然环境下成熟百香果检测及定位模型。针对自然环境下光线以及遮挡的影响,通过MSRCP算法、随机遮挡、数据增扩等图像处理算法对原始数据集进行优化。将优化的数据集输入到YOLOv5网络中训练出最优模型,在检测代码中嵌入双目立体视觉算法。该模型对自然环境下百香果进行检测及成熟度判断,将判断为成熟的百香果进行图像处理,并提取到中心点二维坐标。通过立体匹配及视差计算得到中心点的三维坐标。田间试验结果表明,该模型的目标检测准确率为97.8%,总体准确率为90.2%,平均运行时间为4.85 s。该系统鲁棒性强、实时性好,能够更好地实现自然环境下成熟百香果的检测及定位,为百香果采摘机器人后续工作奠定基础。

Sparse Reconstructive Evidential Clustering for Multi-View Data

Although many multi-view clustering(MVC) algorithms with acceptable performances have been presented, to the best of our knowledge, nearly all of them need to be fed with the correct number of clusters. In addition, these existing algorithms create only the hard and fuzzy partitions for multi-view objects,which are often located in highly-overlapping areas of multi-view feature space. The adoption of hard and fuzzy partition ignores the ambiguity and uncertainty in the assignment of objects, likely leading to performance degradation. To address these issues, we propose a novel sparse reconstructive multi-view evidential clustering algorithm(SRMVEC). Based on a sparse reconstructive procedure, SRMVEC learns a shared affinity matrix across views, and maps multi-view objects to a 2-dimensional humanreadable chart by calculating 2 newly defined mathematical metrics for each object. From this chart, users can detect the number of clusters and select several objects existing in the dataset as cluster centers. Then, SRMVEC derives a credal partition under the framework of evidence theory, improving the fault tolerance of clustering. Ablation studies show the benefits of adopting the sparse reconstructive procedure and evidence theory. Besides,SRMVEC delivers effectiveness on benchmark datasets by outperforming some state-of-the-art methods.

Learning Epipolar Line Window Attention for Stereo Image Super-Resolution Reconstruction

Transformer-based stereo image super-resolution reconstruction(Stereo SR)methods have significantly improved image quality.However,existing methods have deficiencies in paying attention to detailed features and do not consider the offset of pixels along the epipolar lines in complementary views when integrating stereo information.To address these challenges,this paper introduces a novel epipolar line window attention stereo image super-resolution network(EWASSR).For detail feature restoration,we design a feature extractor based on Transformer and convolutional neural network(CNN),which consists of(shifted)window-based self-attention((S)W-MSA)and feature distillation and enhancement blocks(FDEB).This combination effectively solves the problem of global image perception and local feature attention and captures more discriminative high-frequency features of the image.Furthermore,to address the problem of offset of complementary pixels in stereo images,we propose an epipolar line window attention(EWA)mechanism,which divides windows along the epipolar direction to promote efficient matching of shifted pixels,even in pixel smooth areas.More accurate pixel matching can be achieved using adjacent pixels in the window as a reference.Extensive experiments demonstrate that our EWASSR can reconstruct more realistic detailed features.Comparative quantitative results show that in the experimental results of our EWASSR on the Middlebury and Flickr1024 data sets for 2×SR,compared with the recent network,the Peak signal-to-noise ratio(PSNR)increased by 0.37 dB and 0.34 dB,respectively.

Low-Rank Multi-View Subspace Clustering Based on Sparse Regularization

Multi-view Subspace Clustering (MVSC) emerges as an advanced clustering method, designed to integrate diverse views to uncover a common subspace, enhancing the accuracy and robustness of clustering results. The significance of low-rank prior in MVSC is emphasized, highlighting its role in capturing the global data structure across views for improved performance. However, it faces challenges with outlier sensitivity due to its reliance on the Frobenius norm for error measurement. Addressing this, our paper proposes a Low-Rank Multi-view Subspace Clustering Based on Sparse Regularization (LMVSC- Sparse) approach. Sparse regularization helps in selecting the most relevant features or views for clustering while ignoring irrelevant or noisy ones. This leads to a more efficient and effective representation of the data, improving the clustering accuracy and robustness, especially in the presence of outliers or noisy data. By incorporating sparse regularization, LMVSC-Sparse can effectively handle outlier sensitivity, which is a common challenge in traditional MVSC methods relying solely on low-rank priors. Then Alternating Direction Method of Multipliers (ADMM) algorithm is employed to solve the proposed optimization problems. Our comprehensive experiments demonstrate the efficiency and effectiveness of LMVSC-Sparse, offering a robust alternative to traditional MVSC methods.

CNLPA-MVS:Coarse-Hypotheses Guided Non-Local PAtchMatch Multi-View Stereo

In multi-view stereo,unreliable matching in low-textured regions has a negative impact on the completeness of reconstructed models.Since the photometric consistency of low-textured regions is not discriminative under a local window,non-local information provided by the Markov Random Field(MRF)model can alleviate the matching ambiguity but is limited in continuous space with high computational complexity.Owing to its sampling and propagation strategy,PatchMatch multi-view stereo methods have advantages in terms of optimizing the continuous labeling problem.In this paper,we propose a novel method to address this problem,namely the Coarse-Hypotheses Guided Non-Local PAtchMatch Multi-View Stereo(CNLPA-MVS),which takes the advantages of both MRF-based non-local methods and PatchMatch multi-view stereo and compensates for their defects mutually.First,we combine dynamic programing(DP)and sequential propagation along scanlines in parallel to perform CNLPA-MVS,thereby obtaining the optimal depth and normal hypotheses.Second,we introduce coarse inference within a universal window provided by winner-takes-all to eliminate the stripe artifacts caused by DP and improve completeness.Third,we add a local consistency strategy based on the hypotheses of similar color pixels sharing approximate values into CNLPA-MVS for further improving completeness.CNLPA-MVS was validated on public benchmarks and achieved state-of-the-art performance with high completeness.

Practical BRDF reconstruction using reliable geometric regions from multi-view stereo

In this paper,we present a practical method for reconstructing the bidirectional reflectance distribution function(BRDF)from multiple images of a real object composed of a homogeneous material.The key idea is that the BRDF can be sampled after geometry estimation using multi-view stereo(MVS)techniques.Our contribution is selection of reliable samples of lighting,surface normal,and viewing directions for robustness against estimation errors of MVS.Our method is quantitatively evaluated using synthesized images and its effectiveness is shown via real-world experiments.

Multi-View & Transfer Learning for Epilepsy Recognition Based on EEG Signals

Epilepsy is a central nervous system disorder in which brain activity becomes abnormal.Electroencephalogram(EEG)signals,as recordings of brain activity,have been widely used for epilepsy recognition.To study epilep-tic EEG signals and develop artificial intelligence(AI)-assist recognition,a multi-view transfer learning(MVTL-LSR)algorithm based on least squares regression is proposed in this study.Compared with most existing multi-view transfer learning algorithms,MVTL-LSR has two merits:(1)Since traditional transfer learning algorithms leverage knowledge from different sources,which poses a significant risk to data privacy.Therefore,we develop a knowledge transfer mechanism that can protect the security of source domain data while guaranteeing performance.(2)When utilizing multi-view data,we embed view weighting and manifold regularization into the transfer framework to measure the views’strengths and weaknesses and improve generalization ability.In the experimental studies,12 different simulated multi-view&transfer scenarios are constructed from epileptic EEG signals licensed and provided by the Uni-versity of Bonn,Germany.Extensive experimental results show that MVTL-LSR outperforms baselines.The source code will be available on https://github.com/didid5/MVTL-LSR.

ER-Net:Efficient Recalibration Network for Multi-ViewMulti-Person 3D Pose Estimation

Multi-view multi-person 3D human pose estimation is a hot topic in the field of human pose estimation due to its wide range of application scenarios.With the introduction of end-to-end direct regression methods,the field has entered a new stage of development.However,the regression results of joints that are more heavily influenced by external factors are not accurate enough even for the optimal method.In this paper,we propose an effective feature recalibration module based on the channel attention mechanism and a relative optimal calibration strategy,which is applied to themulti-viewmulti-person 3D human pose estimation task to achieve improved detection accuracy for joints that are more severely affected by external factors.Specifically,it achieves relative optimal weight adjustment of joint feature information through the recalibration module and strategy,which enables the model to learn the dependencies between joints and the dependencies between people and their corresponding joints.We call this method as the Efficient Recalibration Network(ER-Net).Finally,experiments were conducted on two benchmark datasets for this task,Campus and Shelf,in which the PCP reached 97.3% and 98.3%,respectively.

Relational graph location network for multi-view image localization

In multi-view image localization task,the features of the images captured from different views should be fused properly.This paper considers the classification-based image localization problem.We propose the relational graph location network(RGLN)to perform this task.In this network,we propose a heterogeneous graph construction approach for graph classification tasks,which aims to describe the location in a more appropriate way,thereby improving the expression ability of the location representation module.Experiments show that the expression ability of the proposed graph construction approach outperforms the compared methods by a large margin.In addition,the proposed localization method outperforms the compared localization methods by around 1.7%in terms of meter-level accuracy.

Diverse Deep Matrix Factorization With Hypergraph Regularization for Multi-View Data Representation

Deep matrix factorization(DMF)has been demonstrated to be a powerful tool to take in the complex hierarchical information of multi-view data(MDR).However,existing multiview DMF methods mainly explore the consistency of multi-view data,while neglecting the diversity among different views as well as the high-order relationships of data,resulting in the loss of valuable complementary information.In this paper,we design a hypergraph regularized diverse deep matrix factorization(HDDMF)model for multi-view data representation,to jointly utilize multi-view diversity and a high-order manifold in a multilayer factorization framework.A novel diversity enhancement term is designed to exploit the structural complementarity between different views of data.Hypergraph regularization is utilized to preserve the high-order geometry structure of data in each view.An efficient iterative optimization algorithm is developed to solve the proposed model with theoretical convergence analysis.Experimental results on five real-world data sets demonstrate that the proposed method significantly outperforms stateof-the-art multi-view learning approaches.

Adaptive Recurrent Iterative Updating Stereo Matching Network

When training a stereo matching network with a single training dataset, the network may overly rely on the learned features of the single training dataset due to differences in the training dataset scenes, resulting in poor performance on all datasets. Therefore, feature consistency between matched pixels is a key factor in solving the network’s generalization ability. To address this issue, this paper proposed a more widely applicable stereo matching network that introduced whitening loss into the feature extraction module of stereo matching, and significantly improved the applicability of the network model by constraining the variation between salient feature pixels. In addition, this paper used a GRU iterative update module in the disparity update calculation stage, which expanded the model’s receptive field at multiple resolutions, allowing for precise disparity estimation not only in rich texture areas but also in low texture areas. The model was trained only on the Scene Flow large-scale dataset, and the disparity estimation was conducted on mainstream datasets such as Middlebury, KITTI 2015, and ETH3D. Compared with earlier stereo matching algorithms, this method not only achieves more accurate disparity estimation but also has wider applicability and stronger robustness.

基于自适应聚合循环递归的稠密点云重建网络

为了解决弱纹理重建难、资源消耗大和重建时间长等问题,提出了一种基于自适应聚合循环递归卷积的多阶段稠密点云重建网络,即A2R2-MVSNet(adaptive aggregation recurrent recursive multi view stereo net)。该方法首先引入一种基于多尺度循环递归残差的特征提取模块,聚合上下文语义信息,以解决弱纹理或无纹理区域特征提取难的问题。在代价体正则化部分,提出一种残差正则化模块,该模块在略微增加内存消耗的前提下,提高了3D CNN提取和聚合上下文语意的能力。实验结果表明,提出的方法在DTU数据集上的综合指标排名靠前,在重建细节上有着更好的体现,且在BlendedMVS数据集上生成了不错的深度图和点云结果,此外网络还在自采集的大规模高分辨率数据集上进行了泛化测试。归功于由粗到细的多阶段思想和我们提出的模块,网络在生成高准确性和完整性深度图的同时,还能进行高分辨率重建以适用于实际问题。

注意力机制与神经渲染的多视图三维重建算法

针对多视图立体网络在弱纹理或非朗伯曲面等挑战性区域重建效果差的问题,首先提出一个基于3个并行扩展卷积和注意力机制的多尺度特征提取模块,在增加感受野的同时捕获特征之间的依赖关系以获取全局上下文信息,从而提升多视图立体网络在挑战性区域特征的表征能力以进行鲁棒的特征匹配。其次在代价体正则化3D CNN部分引入注意力机制,使网络注意于代价体中的重要区域以进行平滑处理。另外建立一个神经渲染网络,该网络利用渲染参考损失精确地解析辐射场景表达的几何外观信息,并引入深度一致性损失保持多视图立体网络与神经渲染网络之间的几何一致性,有效地缓解有噪声代价体对多视图立体网络的不利影响。该算法在室内DTU数据集中测试,点云重建的完整性和整体性指标分别为0.289和0.326,与基准方法CasMVSNet相比,分别提升24.9%和8.2%,即使在挑战性区域也得到高质量的重建效果;在室外Tanks and Temples中级数据集中,点云重建的平均F-score为60.31,与方法UCS-Net相比提升9.9%,体现出较强的泛化能力。

Hawk‐eye‐inspired perception algorithm of stereo vision for obtaining orchard 3D point cloud navigation map

The binocular stereo vision is the lowest cost sensor for obtaining 3D information.Considering the weakness of long‐distance measurement and stability,the improvement of accuracy and stability of stereo vision is urgently required for application of precision agriculture.To address the challenges of stereo vision long‐distance measurement and stable perception without hardware upgrade,inspired by hawk eyes,higher resolution perception and the adaptive HDR(High Dynamic Range)were introduced in this paper.Simulating the function from physiological structure of‘deep fovea’and‘shallow fovea’of hawk eye,the higher resolution reconstruction method in this paper was aimed at ac-curacy improving.Inspired by adjustment of pupils,the adaptive HDR method was proposed for high dynamic range optimisation and stable perception.In various light conditions,compared with default stereo vision,the accuracy of proposed algorithm was improved by 28.0%evaluated by error ratio,and the stability was improved by 26.56%by disparity accuracy.For fixed distance measurement,the maximum improvement was 78.6%by standard deviation.Based on the hawk‐eye‐inspired perception algorithm,the point cloud of orchard was improved both in quality and quantity.The hawk‐eye‐inspired perception algorithm contributed great advance in binocular 3D point cloud recon-struction in orchard navigation map.

语义增强的多视立体视觉方法

针对在基于深度学习技术的特征提取网络中,深层次的卷积神经网络提取的特征缺乏低级语义信息的问题,该文提出了语义增强的多视立体视觉方法。首先,提出了一种ConvLSTM(Convolutional Long Short-Term Memory)语义聚合网络,通过使用ConvLSTM网络结构,对多个卷积层提取的特征图进行预测,得到融合每层语义信息的特征图,有助于在空间上层层抽取图像的高级特征时,利用长短期记忆神经网络结构的记忆功能来增强高层特征图中的低级语义信息,提高了弱纹理区域的重建效果,提高了3D重建的鲁棒性和完整性;其次,提出了一种可见性网络,在灰度图的基础上,通过突出特征图上可见区域的特征,加深了可见区域在特征图中的影响,有助于提高三维重建效果;最后,提取图像的纹理信息,并进入ConvLSTM语义聚合网络提取深层次特征,提高了弱纹理区域的重建效果。与主流的多视立体视觉重建方法相比,重建效果较好。

基于多特征SAD-Census变换的立体匹配算法

视差不连续区域和重复纹理区域的误匹配率高一直是影响双目立体匹配测量精度的主要问题,为此,本文提出一种基于多特征融合的立体匹配算法。首先,在代价计算阶段,通过高斯加权法赋予邻域像素点的权值,从而优化绝对差之和(Sum of Absolute Differences,SAD)算法的计算精度。接着,基于Census变换改进二进制链码方式,将邻域内像素的平均灰度值与梯度图像的灰度均值相融合,进而建立左右图像对应点的判断依据并优化其编码长度。然后,构建基于十字交叉法与改进的引导滤波器相融合的聚合方法,从而实现视差值再分配,以降低误匹配率。最后,通过赢家通吃(Winner Take All,WTA)算法获取初始视差,并采用左右一致性检测方法及亚像素法提高匹配精度,从而获取最终的视差结果。实验结果表明,在Middlebury数据集的测试中,所提SAD-Census算法的平均非遮挡区域和全部区域的误匹配率为分别为2.67%和5.69%,测量200~900 mm距离的平均误差小于2%;而实际三维测量的最大误差为1.5%。实验结果检验了所提算法的有效性和可靠性。

基于分层空间一致性的旋转双目立体校正算法

在旋转双目立体视觉系统中,转台机械间隙导致的左右相机旋转平移偏差,造成立体校正图像的严重畸变。针对该问题,提出一种基于分层空间一致性的旋转双目立体校正算法。采用ORB特征在原始左右图像中进行快速全局立体匹配,设计一种新的特征点全局双层约束,实现匹配点的优选。提出基于内点邻域空间一致性的局部校验方法,实现二次匹配优化,并利用质量排序的优化匹配点集,由八点法基础矩阵估计算法计算左右相机的精确位姿关系,以此完成图像的立体校正。在Oxford和SYNTIM数据集上的典型算法对比实验,验证了所提算法的性能。多角度立体校正实验表明:所提算法可适应光轴夹角变化,在双目最大45°夹角时保证立体校正的质量,匹配点偏差小于0.2像素。

基于Pynq的半全局立体匹配算法实现

立体匹配算法作为双目立体视觉技术的核心算法,具有运算复杂度高、资源占用率高、实时性差的特点。而现有的基于FPGA实现方法多采用高性能FPGA芯片,利用其板卡丰富的资源进行纯verilog语言开发。这样做虽然实时性效果不错,但存在开发难度较大、周期较长、成本较高的弊端。论文针对该问题,提出了一种基于相对低廉的Pynq-z2开发板在PL+PS端协同开发的SGM立体匹配算法,将运算量大的部分在FPGA端进行硬件加速,部分代价优化算法在ARM端利用Python语言实现。实验表明,该算法可有效降低资源占用率,基本满足在相对低廉的嵌入式设备中实现的要求。

电力变压器关键尺寸视觉检测方法及其缩比模型验证

电力变压器的关键尺寸测量是其装配制造、运输安装过程中重要一环,现有测量方法操作繁琐且效率较低。为此,本文提出了一种适用于110 kV油浸式电力变压器关键尺寸视觉检测方法,该方法利用YOLO目标检测算法、Grabcut图像分割算法实现关键组部件的智能检测与分割,然后基于双目立体视觉原理实现套管相间距离及变压器最大截面外形尺寸等关键尺寸的测量。本文搭建了110 kV油浸式电力变压器外观缩比模型,试验分析了拍摄距离和角度等因素对电力变压器关键尺寸视觉检测的影响。结果表明,本文基于缩比模型试验实现了在不同拍摄距离与角度下变压器关键尺寸的视觉检测,验证了该方法的有效性。本文方法可以为110 kV油浸式电力变压器现场尺寸测量提供参考。

一种密集多尺度特征引导代价聚合的改进立体匹配网络

针对目前立体匹配算法在重复纹理、无纹理、边缘等不适定性区域仍存在匹配不准确的问题,提出了一种基于PSMNet的密集多尺度特征引导代价聚合的立体匹配算法—DGNet(Dense multi-scale features Guided aggregation Network)。首先,基于密集连接空洞空间金字塔池化结构设计了密集多尺度特征提取模块,该模块利用不同膨胀率的空洞卷积提取不同尺度的区域级特征,并通过密集连接方式有效整合不同尺度的图像特征,使网络捕获丰富的上下文关系;其次,在每个视差等级下将左右特征图串联形成初始代价体,再提出密集多尺度特征引导代价聚合结构,在聚合代价体的同时自适应融合代价体和密集多尺度特征,从而使后续的解码层在多尺度上下文信息的引导下解码出更加精确和高分辨率的几何信息;最后,将全局优化后的高分辨率代价体送入视差回归模块以获得视差图。实验结果表明:所提算法在KITTI 2015和KITTI 2012数据集上的误匹配率分别降至1.76%和1.24%,SceneFlow数据集上的端点误差降至0.56 px,与GWCNet、CPOP-Net等先进算法相比,所提算法在不适定区域有明显改善。