Learning Epipolar Line Window Attention for Stereo Image Super-Resolution Reconstruction

Transformer-based stereo image super-resolution reconstruction(Stereo SR)methods have significantly improved image quality.However,existing methods have deficiencies in paying attention to detailed features and do not consider the offset of pixels along the epipolar lines in complementary views when integrating stereo information.To address these challenges,this paper introduces a novel epipolar line window attention stereo image super-resolution network(EWASSR).For detail feature restoration,we design a feature extractor based on Transformer and convolutional neural network(CNN),which consists of(shifted)window-based self-attention((S)W-MSA)and feature distillation and enhancement blocks(FDEB).This combination effectively solves the problem of global image perception and local feature attention and captures more discriminative high-frequency features of the image.Furthermore,to address the problem of offset of complementary pixels in stereo images,we propose an epipolar line window attention(EWA)mechanism,which divides windows along the epipolar direction to promote efficient matching of shifted pixels,even in pixel smooth areas.More accurate pixel matching can be achieved using adjacent pixels in the window as a reference.Extensive experiments demonstrate that our EWASSR can reconstruct more realistic detailed features.Comparative quantitative results show that in the experimental results of our EWASSR on the Middlebury and Flickr1024 data sets for 2×SR,compared with the recent network,the Peak signal-to-noise ratio(PSNR)increased by 0.37 dB and 0.34 dB,respectively.

基于Stereo Camera-YOLOv5自然环境下百香果检测与定位模型

针对百香果采摘机器人在自然环境中作业时受复杂光线及遮挡影响,难以快速精确地检测及定位成熟百香果的问题,提出一种基于Stereo Camera-YOLOv5自然环境下成熟百香果检测及定位模型。针对自然环境下光线以及遮挡的影响,通过MSRCP算法、随机遮挡、数据增扩等图像处理算法对原始数据集进行优化。将优化的数据集输入到YOLOv5网络中训练出最优模型,在检测代码中嵌入双目立体视觉算法。该模型对自然环境下百香果进行检测及成熟度判断,将判断为成熟的百香果进行图像处理,并提取到中心点二维坐标。通过立体匹配及视差计算得到中心点的三维坐标。田间试验结果表明,该模型的目标检测准确率为97.8%,总体准确率为90.2%,平均运行时间为4.85 s。该系统鲁棒性强、实时性好,能够更好地实现自然环境下成熟百香果的检测及定位,为百香果采摘机器人后续工作奠定基础。

Hawk‐eye‐inspired perception algorithm of stereo vision for obtaining orchard 3D point cloud navigation map

The binocular stereo vision is the lowest cost sensor for obtaining 3D information.Considering the weakness of long‐distance measurement and stability,the improvement of accuracy and stability of stereo vision is urgently required for application of precision agriculture.To address the challenges of stereo vision long‐distance measurement and stable perception without hardware upgrade,inspired by hawk eyes,higher resolution perception and the adaptive HDR(High Dynamic Range)were introduced in this paper.Simulating the function from physiological structure of‘deep fovea’and‘shallow fovea’of hawk eye,the higher resolution reconstruction method in this paper was aimed at ac-curacy improving.Inspired by adjustment of pupils,the adaptive HDR method was proposed for high dynamic range optimisation and stable perception.In various light conditions,compared with default stereo vision,the accuracy of proposed algorithm was improved by 28.0%evaluated by error ratio,and the stability was improved by 26.56%by disparity accuracy.For fixed distance measurement,the maximum improvement was 78.6%by standard deviation.Based on the hawk‐eye‐inspired perception algorithm,the point cloud of orchard was improved both in quality and quantity.The hawk‐eye‐inspired perception algorithm contributed great advance in binocular 3D point cloud recon-struction in orchard navigation map.

Adaptive Recurrent Iterative Updating Stereo Matching Network

When training a stereo matching network with a single training dataset, the network may overly rely on the learned features of the single training dataset due to differences in the training dataset scenes, resulting in poor performance on all datasets. Therefore, feature consistency between matched pixels is a key factor in solving the network’s generalization ability. To address this issue, this paper proposed a more widely applicable stereo matching network that introduced whitening loss into the feature extraction module of stereo matching, and significantly improved the applicability of the network model by constraining the variation between salient feature pixels. In addition, this paper used a GRU iterative update module in the disparity update calculation stage, which expanded the model’s receptive field at multiple resolutions, allowing for precise disparity estimation not only in rich texture areas but also in low texture areas. The model was trained only on the Scene Flow large-scale dataset, and the disparity estimation was conducted on mainstream datasets such as Middlebury, KITTI 2015, and ETH3D. Compared with earlier stereo matching algorithms, this method not only achieves more accurate disparity estimation but also has wider applicability and stronger robustness.

STEREO卫星的CME观测研究进展

简要回顾利用"日地关系天文台"(Solar Terrestrial Relations Observatory,STEREO)卫星的立体观测资料在日冕物质抛射(Coronal Mass Ejection,CME)研究方面已取得的一些重要进展,主要包括(1)通过极紫外成像仪观测到的日冕极紫外暗化来更准确地估计CME质量,研究CME演化的结构特征;(2)利用STEREO卫星日冕仪的双角度观测,在CME立体传播特征方面取得的新进展;(3)STEREO卫星日球成像仪具有广阔的视场范围,可以跟踪研究CME从太阳表面爆发到形成行星际日冕物质抛射(Interplanetary CME,ICME),及其在内日球层和近地空间的演化特征以及运动特征等。同时,也介绍了利用三角测量技术测定CME特征物理量的新方法。

Hierarchical adaptive stereo matching algorithm for obstacle detection with dynamic programming

An adaptive weighted stereo matching algorithm with multilevel and bidirectional dynamic programming based on ground control points （GCPs） is presented. To decrease time complexity without losing matching precision, using a multilevel search scheme, the coarse matching is processed in typical disparity space image, while the fine matching is processed in disparity-offset space image. In the upper level, GCPs are obtained by enhanced volumetric iterative algorithm enforcing the mutual constraint and the threshold constraint. Under the supervision of the highly reliable GCPs, bidirectional dynamic programming framework is employed to solve the inconsistency in the optimization path. In the lower level, to reduce running time, disparity-offset space is proposed to efficiently achieve the dense disparity image. In addition, an adaptive dual support-weight strategy is presented to aggregate matching cost, which considers photometric and geometric information. Further, post-processing algorithm can ameliorate disparity results in areas with depth discontinuities and related by occlusions using dual threshold algorithm, where missing stereo information is substituted from surrounding regions. To demonstrate the effectiveness of the algorithm, we present the two groups of experimental results for four widely used standard stereo data sets, including discussion on performance and comparison with other methods, which show that the algorithm has not only a fast speed, but also significantly improves the efficiency of holistic optimization.

结合实地观测和STEREO/HI图像观测分析

基于单颗STEREO卫星日球层成像仪(Heliospheric Imager,HI)图像的固定Φ角拟合法(Fixed-Φ,FΦ)和调和均值拟合法(Harmonic-mean,HM),结合STEREO和ACE卫星的太阳风实地观测数据,深入分析了2010年15个日冕物质抛射(CME)事件,对比讨论了这两种方法在提取CME参数如太阳赤道平面的主传播方向、传播速度的效果,其中FΦ拟合法假设CME是固定方向传播的小质点,HM拟合法假设CME为具有球形前沿的通量绳结构,结果发现:(1)使用HM拟合法分析得到的CME主传播方向与太阳-实地观测点的夹角平均值是9.5°,小于FΦ拟合法的19.7°;(2)HM拟合法分析的预计到达时间与实测ICME起始时间的平均误差和最大误差分别为0.282 d和0.805 d,明显小于FΦ拟合法。该文也使用结合STEREO两颗卫星HI图像的直接三角法(Direct-triangulation,DT)和球面切线法(Tangent-to-a-sphere,TS),深入分析了5个朝向地球的CME事件,其中,DT和FΦ拟合法的假设相同,TS和HM拟合法的假设相同,结果发现:(1)这两种方法分析的CME主传播方向与日地连线的夹角最大值分别是13.2°和21.1°,明显小于单颗卫星观测的20.7°和27.5°;(2)其中4个CME事件使用方法得到的线性拟合加速度不超过0.4 m/s2,这说明CME在主传播方向上的速度变化在1 AU内不超过100 km/s;(3)使用TS方法得到的预计到达时间与实测ICME起始时间的绝对误差最小,平均值和最大值分别是2.3 hr和5.8 hr。可见,利用HI图像提取CME传播参数时,加入CME前沿结构假设和结合多角度观测都能够有效地减小拟合误差。

Research on Extraction of Bottom of Shoe Pattern Based on Binocular Stereo Vision

In order to quickly and efficiently get the information of the bottom of the shoe pattern and spraying trajectory, the paper proposes a method based on binocular stereo vision. After acquiring target image, edge detection based on the canny algorithm, the paper begins stereo matching based on area and characteristics of algorithm. To eliminate false matching points, the paper uses the principle of polar geometry in computer vision. For the purpose of gaining the 3D point cloud of spraying curve, the paper adopts the principle of binocular stereo vision 3D measurement, and then carries on cubic spline curve fitting. By HALCON image processing software programming, it proves the feasibility and effectiveness of the method

结合实地观测和STEREO/HI图像观测分析2010年CME事件

本文使用了基于单颗STEREO卫星日球层成像仪(Heliospheric Imager,HI)图像的固定Φ角拟合法(Fixed-Φ,FΦ)和调和均值拟合法(Harmonic-mean,HM),结合STEREO和ACE卫星的太阳风实地观测数据,深入分析了2010年15个日冕物质抛射(CME)事件,对比讨论了这两种方法在提取CME参数如太阳赤道平面的主传播方向、传播速度的效果,其中FΦ拟合法假设CME是固定方向传播的小质点,HM拟合法假设CME为具有球形前沿的通量绳结构,结果发现:(1)使用HM拟合法分析得到的CME主传播方向与太阳-实地观测点的夹角平均值是9.5°,小于FΦ拟合法的19.7°;(2)HM拟合法分析的预计到达时间与实测ICME起始时间的平均误差和最大误差分别为0.282天和0.805天,明显小于FΦ拟合法.本文也使用结合STEREO两颗卫星HI图像的直接三角法(Direct-triangulation,DT)和球面切线法(Tangent-to-a-sphere,TS),深入分析了5个朝向地球的CME事件,其中,DT和FΦ拟合法的假设相同,TS和HM拟合法的假设相同,结果发现:(1)这两种方法分析的CME主传播方向与日地连线的夹角最大值分别是13.2°和21.1°,明显小于单颗卫星观测的20.7°和27.5°;(2)其中4个CME事件使用方法得到的线性拟合加速度不超过0.4m.s-2,这说明CME在主传播方向上的速度变化在1AU内不超过100km.s-1;(3)使用TS方法得到的预计到达时间与实测ICME起始时间的绝对误差最小,平均值和最大值分别是2.3h和5.8h.可见,利用HI图像提取CME传播参数时,加入CME前沿结构假设和结合多角度观测都能够有效地减小拟合误差.

ACE和STEREO-B太阳风速度的统计分析

采用小波分析方法讨论了2008--2011年ACE和STEREO—B卫星太阳风速度的27天周期特性，运用一元线性回归分析方法计算了两颗卫星太阳风速度的线性表达式，分析和计算结果表明，ACE和STEREO—B卫星探测到的太阳风速均在8-16d，16-32d的频域尺度上较为显著，在局部相同时域上，二者太阳风速的27天周期均较为显著；当太阳活动为低年时，相关性好．作为应用实例，使用STEREO—B卫星太阳风探测数据，预测2012年10月1-17日的ACE太阳风速度，结果表明，预测值与实测值趋势一致，CME过程对预测值有一定影响．通过本项研究，初步统计出了ACE与STEREO-B太阳风速度的关系，利用STEREO—B能够提前数天监测到即将由太阳吹向地球的太阳风特性，为建立直观的重现型地磁暴中期预报模型奠定了基础．

植物立体可视化模拟系统原型StereoPlants的构建

随着虚拟植物可视化研究的深入,植物群体以及复杂的植物器官或微观显微结构模拟已成为趋势。目前植物模拟中可视化结果展示方式多是对渲染的图像进行平面显示,不便于对大场景植物群体或复杂微观显微结构可视化结果的认知。为此,该文提出并实现了一个立体显示植物模拟的系统原型,该系统完成植物可视化立体内容的生成到立体展现,系统的软件部分基于跨平台的FLTK界面库与OpenGL图形库完成植物模拟立体可视化架构,支持对现有的基于该图形库的可视化模拟模型的移植,硬件部分由计算机与被动式立体展示系统组成。为了测试系统的可行性与有效性,在立体可视化框架上移植现有植物根系可视化模拟,结果显示,该立体显示植物模拟系统原型可生动地、沉浸地立体显示植物可视化模拟。该原型系统使得植物群体以及复杂的植物器官或微观显微结构的可视化更直观、更容易被认知。

6-DOF MOTION AND CENTER OF ROTATION ESTIMATION BASED ON STEREO VISION

A new motion model and estimation algorithm is proposed to compute the general rigid motion object＇s 6-DOF motion parameters and center of rotation based on stereo vision. The object＇s 6-DOF motion model is designed from the rigid object＇s motion character under the two defined reference frames. According to the rigid object＇s motion model and motion dynamics knowledge, the corresponding motion algorithm to compute the 6-DOF motion parameters is worked out. By the rigid object pure rotation motion model and space sphere geometry knowledge, the center of rotation may be calculated after eliminating the translation motion out of the 6-DOF motion. The motion equations are educed based on the motion model and the closed-form solutions are figured out. To heighten the motion estimation algorithm＇s robust, RANSAC algorithm is applied to delete the outliers. Simulation and real experiments are conducted and the experiment results are analyzed. The results prove the motion model＇s correction and algorithm＇s validity.

Automatic three-dimensional reconstruction based on four-view stereo vision using checkerboard pattern

An automatic three-dimensional(3D) reconstruction method based on four-view stereo vision using checkerboard pattern is presented. Mismatches easily exist in traditional binocular stereo matching due to the repeatable or similar features of binocular images. In order to reduce the probability of mismatching and improve the measure precision, a four-camera measurement system which can add extra matching constraints and offer multiple measurements is applied in this work. Moreover, a series of different checkerboard patterns are projected onto the object to obtain dense feature points and remove mismatched points. Finally, the 3D model is generated by performing Delaunay triangulation and texture mapping on the point cloud obtained by four-view matching. This method was tested on the 3D reconstruction of a terracotta soldier sculpture and the Buddhas in the Mogao Grottoes. Their point clouds without mismatched points were obtained and less processing time was consumed in most cases relative to binocular matching. These good reconstructed models show the effectiveness of the method.

Geometrical-Analysis-Based Algorithm for Stereo Matching of Single-Lens Binocular and Multi-Ocular Stereovision System

A geometrical analysis based algorithm is proposed to achieve the stereo matching of a single-lens prism based stereovision system. By setting the multi- face prism in frontal position of the static CCD （CM-140MCL） camera, equivalent stereo images with different orientations are captured synchronously by virtual cameras which are defined by two boundary lines： the optical axis and CCD camera field of view boundary. Subsequently, the geometrical relationship between the 2D stereo images and corresponding 3D scene is established by employing two fundamentals： ray sketching in which all the pertinent points, lines, and planes are expressed in the 3D camera coordinates and the rule of refraction. Landing on this relationship, the epipolar geometry is thus obtained by fitting a set of corresponding candidate points and thereafter, stereo matching of the prism based stereovision system is obtained. Moreover, the unique geometrical properties of the imaging system allow the proposed method free from the complicated camera calibration procedures and to be easily generalized from binocular and tri-oeular to multi-ocular stereovision systems. The performance of the algorithm is presented through the experiments on the binocular imaging system and the comparison with a conventional projection method demonstrates the efficient assessment of our novel contributions.

Performance Analysis and Evaluation of Geometric Parameters in Stereo Deflectometry

This paper presents a novel geometric parameters analysis to improve the measurement accuracy of stereo deflectometry.Stereo deflectometry can be used to obtain form information for freeform specular surfaces.A measurement system based on stereo deflectometry typically consists of a fringe-displaying screen,a main camera,and a reference camera.The arrangement of the components of a stereo deflectometry system is important for achieving high-accuracy measurements.In this paper,four geometric parameters of a stereo deflectometry system are analyzed and evaluated:the distance between the main camera and the measured object surface,the angle between the main camera ray and the surface normal,the distance between the fringe-displaying screen and the object,and the angle between the main camera and the reference camera.The influence of the geometric parameters on the measurement accuracy is evaluated.Experiments are performed using simulated and experimental data.The experimental results confirm the impact of these parameters on the measurement accuracy.A measurement system based on the proposed analysis has been set up to measure a stock concave mirror.Through a comparison of the given surface parameters of the concave mirror,a global measurement accuracy of 154.2 nm was achieved.

Saliency detection based on superpixels clustering and stereo disparity

Reliable saliency detection can be used to quickly and effectively locate objects in images. In this paper, a novel algorithm for saliency detection based on superpixels clustering and stereo disparity （SDC） is proposed. Firstly, we use an improved superpixels clustering method to decompose the given image. Then, the disparity of each superpixel is computed by a modified stereo correspondence algorithm. Finally, a new measure which combines stereo disparity with color contrast and spatial coherence is defined to evaluate the saliency of each superpixel. From the experiments we can see that regions with high disparity can get higher saliency value, and the saliency maps have the same resolution with the source images, objects in the map have clear boundaries. Due to the use of superpixel and stereo disparity information, the proposed method is computationally efficient and outperforms some state-of-the-art color- based saliency detection methods.

Accuracy study of a binocular-stereo-vision-based navigation robot for minimally invasive interventional procedures

BACKGROUND Medical robot is a promising surgical tool,but no specific one has been designed for interventional treatment of chronic pain.We developed a computed tomography-image based navigation robot using a new registration method with binocular vision.This kind of robot is appropriate for minimal invasive interventional procedures and easy to operate.The feasibility,accuracy and stability of this new robot need to be tested.AIM To assess quantitatively the feasibility,accuracy and stability of the binocularstereo-vision-based navigation robot for minimally invasive interventional procedures.METHODS A box model was designed for assessing the accuracy for targets at different distances.Nine(three sets)lead spheres were embedded in the model as puncture goals.The entry-to-target distances were set 50 mm(short-distance),100 mm(medium-distance)and 150 mm(long-distance).Puncture procedure was repeated three times for each goal.The Euclidian error of each puncture was calculated and statistically analyzed.Three head phantoms were used to explore the clinical feasibility and stability.Three independent operators conducted foramen ovale placement on head phantoms(both sides)by freehand or under the guidance of robot(18 punctures with each method).The operation time,adjustment time and one-time success rate were recorded,and the two guidancemethods were compared.RESULTS On the box model,the mean puncture errors of navigation robot were 1.7±0.9 mm for the short-distance target,2.4±1.0 mm for the moderate target and 4.4±1.4 mm for the long-distance target.On the head phantom,no obvious differences in operation time and adjustment time were found among the three performers(P>0.05).The median adjustment time was significantly less under the guidance of the robot than under free hand.The one-time success rate was significantly higher with the robot(P<0.05).There was no obvious difference in operation time between the two methods(P>0.05).CONCLUSION In the laboratory environment,accuracy of binocular-stereo-vision-based navigation robot is acceptable for target at 100 mm depth or less.Compared with freehand,foramen ovale placement accuracy can be improved with robot guidance.

Fast Adaptive Support-Weight Stereo Matching Algorithm

Adaptive support-weight (ASW) stereo matching algorithm is widely used in the field of three-dimensional (3D) reconstruction owing to its relatively high matching accuracy.However,since all the weight coefficients need to be calculated in the whole disparity range for each pixel,the algorithm is extremely time-consuming.To solve this problem,a fast ASW algorithm is proposed using twice aggregation.First,a novel weight coefficient which adapts cosine function to satisfy the weight distribution discipline is proposed to accomplish the first cost aggregation.Then,the disparity range is divided into several sub-ranges and local optimal disparities are selected from each of them.For each pixel,only the ASW at the location of local optimal disparities is calculated,and thus,the complexity of the algorithm is greatly reduced.Experimental results show that the proposed algorithm can reduce the amount of calculation by 70% and improve the matching accuracy by 6% for the 15 images on Middlebury Website on average.

Robot stereo vision calibration method with genetic algorithm and particle swarm optimization

Accurate stereo vision calibration is a preliminary step towards high-precision visual posi- tioning of robot. Combining with the characteristics of genetic algorithm （GA） and particle swarm optimization （PSO）, a three-stage calibration method based on hybrid intelligent optimization is pro- posed for nonlinear camera models in this paper. The motivation is to improve the accuracy of the calibration process. In this approach, the stereo vision calibration is considered as an optimization problem that can be solved by the GA and PSO. The initial linear values can be obtained in the frost stage. Then in the second stage, two cameras＇ parameters are optimized separately. Finally, the in- tegrated optimized calibration of two models is obtained in the third stage. Direct linear transforma- tion （DLT）, GA and PSO are individually used in three stages. It is shown that the results of every stage can correctly find near-optimal solution and it can be used to initialize the next stage. Simula- tion analysis and actual experimental results indicate that this calibration method works more accu- rate and robust in noisy environment compared with traditional calibration methods. The proposed method can fulfill the requirements of robot sophisticated visual operation.

Maintenance and Management Model for the Urban Stereo Landscape Forest in Chongqing Academy of Science and Technology

Based on landscaping practices in the stereo landscape demonstration project of Chongqing Academy of Science and Technology (CAST),current management and maintenance of the stereo landscapes were introduced by analyzing maintenance and management of the plants in the stereo landscape forest.The maintenance and management models were summarized according to the management styles,concrete measures and new technologies were studied,laws and key points for the maintenance and management in different seasons were given,and finally deficiencies and solutions were proposed to improve the future maintenance and management.