基于Stereo Camera-YOLOv5自然环境下百香果检测与定位模型

针对百香果采摘机器人在自然环境中作业时受复杂光线及遮挡影响,难以快速精确地检测及定位成熟百香果的问题,提出一种基于Stereo Camera-YOLOv5自然环境下成熟百香果检测及定位模型。针对自然环境下光线以及遮挡的影响,通过MSRCP算法、随机遮挡、数据增扩等图像处理算法对原始数据集进行优化。将优化的数据集输入到YOLOv5网络中训练出最优模型,在检测代码中嵌入双目立体视觉算法。该模型对自然环境下百香果进行检测及成熟度判断,将判断为成熟的百香果进行图像处理,并提取到中心点二维坐标。通过立体匹配及视差计算得到中心点的三维坐标。田间试验结果表明,该模型的目标检测准确率为97.8%,总体准确率为90.2%,平均运行时间为4.85 s。该系统鲁棒性强、实时性好,能够更好地实现自然环境下成熟百香果的检测及定位,为百香果采摘机器人后续工作奠定基础。

Sparse Reconstructive Evidential Clustering for Multi-View Data

Although many multi-view clustering(MVC) algorithms with acceptable performances have been presented, to the best of our knowledge, nearly all of them need to be fed with the correct number of clusters. In addition, these existing algorithms create only the hard and fuzzy partitions for multi-view objects,which are often located in highly-overlapping areas of multi-view feature space. The adoption of hard and fuzzy partition ignores the ambiguity and uncertainty in the assignment of objects, likely leading to performance degradation. To address these issues, we propose a novel sparse reconstructive multi-view evidential clustering algorithm(SRMVEC). Based on a sparse reconstructive procedure, SRMVEC learns a shared affinity matrix across views, and maps multi-view objects to a 2-dimensional humanreadable chart by calculating 2 newly defined mathematical metrics for each object. From this chart, users can detect the number of clusters and select several objects existing in the dataset as cluster centers. Then, SRMVEC derives a credal partition under the framework of evidence theory, improving the fault tolerance of clustering. Ablation studies show the benefits of adopting the sparse reconstructive procedure and evidence theory. Besides,SRMVEC delivers effectiveness on benchmark datasets by outperforming some state-of-the-art methods.

Contrastive Consistency and Attentive Complementarity for Deep Multi-View Subspace Clustering

Deep multi-view subspace clustering (DMVSC) based on self-expression has attracted increasing attention dueto its outstanding performance and nonlinear application. However, most existing methods neglect that viewprivatemeaningless information or noise may interfere with the learning of self-expression, which may lead to thedegeneration of clustering performance. In this paper, we propose a novel framework of Contrastive Consistencyand Attentive Complementarity (CCAC) for DMVsSC. CCAC aligns all the self-expressions of multiple viewsand fuses them based on their discrimination, so that it can effectively explore consistent and complementaryinformation for achieving precise clustering. Specifically, the view-specific self-expression is learned by a selfexpressionlayer embedded into the auto-encoder network for each view. To guarantee consistency across views andreduce the effect of view-private information or noise, we align all the view-specific self-expressions by contrastivelearning. The aligned self-expressions are assigned adaptive weights by channel attention mechanism according totheir discrimination. Then they are fused by convolution kernel to obtain consensus self-expression withmaximumcomplementarity ofmultiple views. Extensive experimental results on four benchmark datasets and one large-scaledataset of the CCAC method outperformother state-of-the-artmethods, demonstrating its clustering effectiveness.

Learning Epipolar Line Window Attention for Stereo Image Super-Resolution Reconstruction

Transformer-based stereo image super-resolution reconstruction(Stereo SR)methods have significantly improved image quality.However,existing methods have deficiencies in paying attention to detailed features and do not consider the offset of pixels along the epipolar lines in complementary views when integrating stereo information.To address these challenges,this paper introduces a novel epipolar line window attention stereo image super-resolution network(EWASSR).For detail feature restoration,we design a feature extractor based on Transformer and convolutional neural network(CNN),which consists of(shifted)window-based self-attention((S)W-MSA)and feature distillation and enhancement blocks(FDEB).This combination effectively solves the problem of global image perception and local feature attention and captures more discriminative high-frequency features of the image.Furthermore,to address the problem of offset of complementary pixels in stereo images,we propose an epipolar line window attention(EWA)mechanism,which divides windows along the epipolar direction to promote efficient matching of shifted pixels,even in pixel smooth areas.More accurate pixel matching can be achieved using adjacent pixels in the window as a reference.Extensive experiments demonstrate that our EWASSR can reconstruct more realistic detailed features.Comparative quantitative results show that in the experimental results of our EWASSR on the Middlebury and Flickr1024 data sets for 2×SR,compared with the recent network,the Peak signal-to-noise ratio(PSNR)increased by 0.37 dB and 0.34 dB,respectively.

Warhead fragments motion trajectories tracking and spatio-temporal distribution reconstruction method based on high-speed stereo photography

High speed photography technique is potentially the most effective way to measure the motion parameter of warhead fragment benefiting from its advantages of high accuracy,high resolution and high efficiency.However,it faces challenge in dense objects tracking and 3D trajectories reconstruction due to the characteristics of small size and dense distribution of fragment swarm.To address these challenges,this work presents a warhead fragments motion trajectories tracking and spatio-temporal distribution reconstruction method based on high-speed stereo photography.Firstly,background difference algorithm is utilized to extract the center and area of each fragment in the image sequence.Subsequently,a multi-object tracking(MOT)algorithm using Kalman filtering and Hungarian optimal assignment is developed to realize real-time and robust trajectories tracking of fragment swarm.To reconstruct 3D motion trajectories,a global stereo trajectories matching strategy is presented,which takes advantages of epipolar constraint and continuity constraint to correctly retrieve stereo correspondence followed by 3D trajectories refinement using polynomial fitting.Finally,the simulation and experimental results demonstrate that the proposed method can accurately track the motion trajectories and reconstruct the spatio-temporal distribution of 1.0×10^(3)fragments in a field of view(FOV)of 3.2 m×2.5 m,and the accuracy of the velocity estimation can achieve 98.6%.

Low-Rank Multi-View Subspace Clustering Based on Sparse Regularization

Multi-view Subspace Clustering (MVSC) emerges as an advanced clustering method, designed to integrate diverse views to uncover a common subspace, enhancing the accuracy and robustness of clustering results. The significance of low-rank prior in MVSC is emphasized, highlighting its role in capturing the global data structure across views for improved performance. However, it faces challenges with outlier sensitivity due to its reliance on the Frobenius norm for error measurement. Addressing this, our paper proposes a Low-Rank Multi-view Subspace Clustering Based on Sparse Regularization (LMVSC- Sparse) approach. Sparse regularization helps in selecting the most relevant features or views for clustering while ignoring irrelevant or noisy ones. This leads to a more efficient and effective representation of the data, improving the clustering accuracy and robustness, especially in the presence of outliers or noisy data. By incorporating sparse regularization, LMVSC-Sparse can effectively handle outlier sensitivity, which is a common challenge in traditional MVSC methods relying solely on low-rank priors. Then Alternating Direction Method of Multipliers (ADMM) algorithm is employed to solve the proposed optimization problems. Our comprehensive experiments demonstrate the efficiency and effectiveness of LMVSC-Sparse, offering a robust alternative to traditional MVSC methods.

CNLPA-MVS:Coarse-Hypotheses Guided Non-Local PAtchMatch Multi-View Stereo

In multi-view stereo,unreliable matching in low-textured regions has a negative impact on the completeness of reconstructed models.Since the photometric consistency of low-textured regions is not discriminative under a local window,non-local information provided by the Markov Random Field(MRF)model can alleviate the matching ambiguity but is limited in continuous space with high computational complexity.Owing to its sampling and propagation strategy,PatchMatch multi-view stereo methods have advantages in terms of optimizing the continuous labeling problem.In this paper,we propose a novel method to address this problem,namely the Coarse-Hypotheses Guided Non-Local PAtchMatch Multi-View Stereo(CNLPA-MVS),which takes the advantages of both MRF-based non-local methods and PatchMatch multi-view stereo and compensates for their defects mutually.First,we combine dynamic programing(DP)and sequential propagation along scanlines in parallel to perform CNLPA-MVS,thereby obtaining the optimal depth and normal hypotheses.Second,we introduce coarse inference within a universal window provided by winner-takes-all to eliminate the stripe artifacts caused by DP and improve completeness.Third,we add a local consistency strategy based on the hypotheses of similar color pixels sharing approximate values into CNLPA-MVS for further improving completeness.CNLPA-MVS was validated on public benchmarks and achieved state-of-the-art performance with high completeness.

Practical BRDF reconstruction using reliable geometric regions from multi-view stereo

In this paper,we present a practical method for reconstructing the bidirectional reflectance distribution function(BRDF)from multiple images of a real object composed of a homogeneous material.The key idea is that the BRDF can be sampled after geometry estimation using multi-view stereo(MVS)techniques.Our contribution is selection of reliable samples of lighting,surface normal,and viewing directions for robustness against estimation errors of MVS.Our method is quantitatively evaluated using synthesized images and its effectiveness is shown via real-world experiments.

Hawk‐eye‐inspired perception algorithm of stereo vision for obtaining orchard 3D point cloud navigation map

The binocular stereo vision is the lowest cost sensor for obtaining 3D information.Considering the weakness of long‐distance measurement and stability,the improvement of accuracy and stability of stereo vision is urgently required for application of precision agriculture.To address the challenges of stereo vision long‐distance measurement and stable perception without hardware upgrade,inspired by hawk eyes,higher resolution perception and the adaptive HDR(High Dynamic Range)were introduced in this paper.Simulating the function from physiological structure of‘deep fovea’and‘shallow fovea’of hawk eye,the higher resolution reconstruction method in this paper was aimed at ac-curacy improving.Inspired by adjustment of pupils,the adaptive HDR method was proposed for high dynamic range optimisation and stable perception.In various light conditions,compared with default stereo vision,the accuracy of proposed algorithm was improved by 28.0%evaluated by error ratio,and the stability was improved by 26.56%by disparity accuracy.For fixed distance measurement,the maximum improvement was 78.6%by standard deviation.Based on the hawk‐eye‐inspired perception algorithm,the point cloud of orchard was improved both in quality and quantity.The hawk‐eye‐inspired perception algorithm contributed great advance in binocular 3D point cloud recon-struction in orchard navigation map.

Multi-View & Transfer Learning for Epilepsy Recognition Based on EEG Signals

Epilepsy is a central nervous system disorder in which brain activity becomes abnormal.Electroencephalogram(EEG)signals,as recordings of brain activity,have been widely used for epilepsy recognition.To study epilep-tic EEG signals and develop artificial intelligence(AI)-assist recognition,a multi-view transfer learning(MVTL-LSR)algorithm based on least squares regression is proposed in this study.Compared with most existing multi-view transfer learning algorithms,MVTL-LSR has two merits:(1)Since traditional transfer learning algorithms leverage knowledge from different sources,which poses a significant risk to data privacy.Therefore,we develop a knowledge transfer mechanism that can protect the security of source domain data while guaranteeing performance.(2)When utilizing multi-view data,we embed view weighting and manifold regularization into the transfer framework to measure the views’strengths and weaknesses and improve generalization ability.In the experimental studies,12 different simulated multi-view&transfer scenarios are constructed from epileptic EEG signals licensed and provided by the Uni-versity of Bonn,Germany.Extensive experimental results show that MVTL-LSR outperforms baselines.The source code will be available on https://github.com/didid5/MVTL-LSR.

ER-Net:Efficient Recalibration Network for Multi-ViewMulti-Person 3D Pose Estimation

Multi-view multi-person 3D human pose estimation is a hot topic in the field of human pose estimation due to its wide range of application scenarios.With the introduction of end-to-end direct regression methods,the field has entered a new stage of development.However,the regression results of joints that are more heavily influenced by external factors are not accurate enough even for the optimal method.In this paper,we propose an effective feature recalibration module based on the channel attention mechanism and a relative optimal calibration strategy,which is applied to themulti-viewmulti-person 3D human pose estimation task to achieve improved detection accuracy for joints that are more severely affected by external factors.Specifically,it achieves relative optimal weight adjustment of joint feature information through the recalibration module and strategy,which enables the model to learn the dependencies between joints and the dependencies between people and their corresponding joints.We call this method as the Efficient Recalibration Network(ER-Net).Finally,experiments were conducted on two benchmark datasets for this task,Campus and Shelf,in which the PCP reached 97.3% and 98.3%,respectively.

Relational graph location network for multi-view image localization

In multi-view image localization task,the features of the images captured from different views should be fused properly.This paper considers the classification-based image localization problem.We propose the relational graph location network(RGLN)to perform this task.In this network,we propose a heterogeneous graph construction approach for graph classification tasks,which aims to describe the location in a more appropriate way,thereby improving the expression ability of the location representation module.Experiments show that the expression ability of the proposed graph construction approach outperforms the compared methods by a large margin.In addition,the proposed localization method outperforms the compared localization methods by around 1.7%in terms of meter-level accuracy.

Diverse Deep Matrix Factorization With Hypergraph Regularization for Multi-View Data Representation

Deep matrix factorization(DMF)has been demonstrated to be a powerful tool to take in the complex hierarchical information of multi-view data(MDR).However,existing multiview DMF methods mainly explore the consistency of multi-view data,while neglecting the diversity among different views as well as the high-order relationships of data,resulting in the loss of valuable complementary information.In this paper,we design a hypergraph regularized diverse deep matrix factorization(HDDMF)model for multi-view data representation,to jointly utilize multi-view diversity and a high-order manifold in a multilayer factorization framework.A novel diversity enhancement term is designed to exploit the structural complementarity between different views of data.Hypergraph regularization is utilized to preserve the high-order geometry structure of data in each view.An efficient iterative optimization algorithm is developed to solve the proposed model with theoretical convergence analysis.Experimental results on five real-world data sets demonstrate that the proposed method significantly outperforms stateof-the-art multi-view learning approaches.

Adaptive Recurrent Iterative Updating Stereo Matching Network

When training a stereo matching network with a single training dataset, the network may overly rely on the learned features of the single training dataset due to differences in the training dataset scenes, resulting in poor performance on all datasets. Therefore, feature consistency between matched pixels is a key factor in solving the network’s generalization ability. To address this issue, this paper proposed a more widely applicable stereo matching network that introduced whitening loss into the feature extraction module of stereo matching, and significantly improved the applicability of the network model by constraining the variation between salient feature pixels. In addition, this paper used a GRU iterative update module in the disparity update calculation stage, which expanded the model’s receptive field at multiple resolutions, allowing for precise disparity estimation not only in rich texture areas but also in low texture areas. The model was trained only on the Scene Flow large-scale dataset, and the disparity estimation was conducted on mainstream datasets such as Middlebury, KITTI 2015, and ETH3D. Compared with earlier stereo matching algorithms, this method not only achieves more accurate disparity estimation but also has wider applicability and stronger robustness.

基于自适应聚合循环递归的稠密点云重建网络

为了解决弱纹理重建难、资源消耗大和重建时间长等问题,提出了一种基于自适应聚合循环递归卷积的多阶段稠密点云重建网络,即A2R2-MVSNet(adaptive aggregation recurrent recursive multi view stereo net)。该方法首先引入一种基于多尺度循环递归残差的特征提取模块,聚合上下文语义信息,以解决弱纹理或无纹理区域特征提取难的问题。在代价体正则化部分,提出一种残差正则化模块,该模块在略微增加内存消耗的前提下,提高了3D CNN提取和聚合上下文语意的能力。实验结果表明,提出的方法在DTU数据集上的综合指标排名靠前,在重建细节上有着更好的体现,且在BlendedMVS数据集上生成了不错的深度图和点云结果,此外网络还在自采集的大规模高分辨率数据集上进行了泛化测试。归功于由粗到细的多阶段思想和我们提出的模块,网络在生成高准确性和完整性深度图的同时,还能进行高分辨率重建以适用于实际问题。

注意力机制与神经渲染的多视图三维重建算法

针对多视图立体网络在弱纹理或非朗伯曲面等挑战性区域重建效果差的问题,首先提出一个基于3个并行扩展卷积和注意力机制的多尺度特征提取模块,在增加感受野的同时捕获特征之间的依赖关系以获取全局上下文信息,从而提升多视图立体网络在挑战性区域特征的表征能力以进行鲁棒的特征匹配。其次在代价体正则化3D CNN部分引入注意力机制,使网络注意于代价体中的重要区域以进行平滑处理。另外建立一个神经渲染网络,该网络利用渲染参考损失精确地解析辐射场景表达的几何外观信息,并引入深度一致性损失保持多视图立体网络与神经渲染网络之间的几何一致性,有效地缓解有噪声代价体对多视图立体网络的不利影响。该算法在室内DTU数据集中测试,点云重建的完整性和整体性指标分别为0.289和0.326,与基准方法CasMVSNet相比,分别提升24.9%和8.2%,即使在挑战性区域也得到高质量的重建效果;在室外Tanks and Temples中级数据集中,点云重建的平均F-score为60.31,与方法UCS-Net相比提升9.9%,体现出较强的泛化能力。

语义增强的多视立体视觉方法

针对在基于深度学习技术的特征提取网络中,深层次的卷积神经网络提取的特征缺乏低级语义信息的问题,该文提出了语义增强的多视立体视觉方法。首先,提出了一种ConvLSTM(Convolutional Long Short-Term Memory)语义聚合网络,通过使用ConvLSTM网络结构,对多个卷积层提取的特征图进行预测,得到融合每层语义信息的特征图,有助于在空间上层层抽取图像的高级特征时,利用长短期记忆神经网络结构的记忆功能来增强高层特征图中的低级语义信息,提高了弱纹理区域的重建效果,提高了3D重建的鲁棒性和完整性;其次,提出了一种可见性网络,在灰度图的基础上,通过突出特征图上可见区域的特征,加深了可见区域在特征图中的影响,有助于提高三维重建效果;最后,提取图像的纹理信息,并进入ConvLSTM语义聚合网络提取深层次特征,提高了弱纹理区域的重建效果。与主流的多视立体视觉重建方法相比,重建效果较好。

基于特征优化的Census立体匹配方法

针对传统Census立体匹配算法在弱纹理和边缘区域匹配精度较差的问题,提出一种基于特征信息优化的代价计算方法,在窗口中融入更多的差异信息以获得更精确的像素视差值。随后采用多方向路径独立的线扫描优化计算聚合代价以进一步提高匹配精度。为获得更好的遮挡区域匹配效果,提出一种基于差异填充的视差优化方法,对遮挡像素进行识别和视差填充。为提高算法的效率,提出一种基于降采样策略的算法运行模式,通过缩小视差搜索范围以减少硬件负荷。最后以五组标准图像为输入进行改进Census算法性能检验,结果显示,平均误匹配率为6.12%,较改进前降低了2.45%,算法效率平均提升17.7%。

基于立体视觉下联合收获机边缘检测技术的研究

智能联合收割机是目前作物收获的发展热点与研究重点技术之一,而确定田间地块边缘、未收割作物边缘、联合收获机直线行驶和在作物地头转向是实现全田间自动收获的基础。为此,设计了一种基于立体视觉下多种作物收获边缘快速检测方法,保持作物收获边缘在检测目标区域内,并提出了一种基于HSV(色调、饱和度和值)空间扫描的动态感兴趣区域提取算法,利用Ostu算法获得作物未收获面积,实现未收获作物边缘和作物末端边缘的同时检测,并根据作物收获末端边缘自动转向到下一个收割路径。田间验证试验表明:水稻检测准确率高于98%,玉米的检测准确率高于94%,平均处理速度为49frame/ms。研究结果表明:提出的基于立体视觉下联合收获机边缘检测及精准转向系统具有较高的工作精度和效率,可为提高作物收获性能提供技术参考与借鉴。

无人机立体视觉识别船舶航行风险仿真

由于船舶交通受碰撞、搁浅、走锚等随机干扰因素的影响,导致对船舶航行目标的跟踪与风险预警具有较大难度。为增强海上航行的安全性,提出一种基于无人机立体视觉的船舶航行风险识别方法。引入双目视觉立体技术,获得目标船舶在航行过程中的图像数据。利用卡尔曼预测器优化连续性自适应均值漂移算法,跟踪目标船舶,通过高斯混合模型识别船舶航行风险。实验结果表明,研究方法对不同海域的船舶跟踪曲线与船舶实际航行曲线具有较高拟合度,且上述方法的船舶航行风险识别正确率高于90%,且误报率低于0.2%,说明提出方法的应用可靠性较高。