Hawk‐eye‐inspired perception algorithm of stereo vision for obtaining orchard 3D point cloud navigation map

The binocular stereo vision is the lowest cost sensor for obtaining 3D information.Considering the weakness of long‐distance measurement and stability,the improvement of accuracy and stability of stereo vision is urgently required for application of precision agriculture.To address the challenges of stereo vision long‐distance measurement and stable perception without hardware upgrade,inspired by hawk eyes,higher resolution perception and the adaptive HDR(High Dynamic Range)were introduced in this paper.Simulating the function from physiological structure of‘deep fovea’and‘shallow fovea’of hawk eye,the higher resolution reconstruction method in this paper was aimed at ac-curacy improving.Inspired by adjustment of pupils,the adaptive HDR method was proposed for high dynamic range optimisation and stable perception.In various light conditions,compared with default stereo vision,the accuracy of proposed algorithm was improved by 28.0%evaluated by error ratio,and the stability was improved by 26.56%by disparity accuracy.For fixed distance measurement,the maximum improvement was 78.6%by standard deviation.Based on the hawk‐eye‐inspired perception algorithm,the point cloud of orchard was improved both in quality and quantity.The hawk‐eye‐inspired perception algorithm contributed great advance in binocular 3D point cloud recon-struction in orchard navigation map.

6-DOF MOTION AND CENTER OF ROTATION ESTIMATION BASED ON STEREO VISION

A new motion model and estimation algorithm is proposed to compute the general rigid motion object＇s 6-DOF motion parameters and center of rotation based on stereo vision. The object＇s 6-DOF motion model is designed from the rigid object＇s motion character under the two defined reference frames. According to the rigid object＇s motion model and motion dynamics knowledge, the corresponding motion algorithm to compute the 6-DOF motion parameters is worked out. By the rigid object pure rotation motion model and space sphere geometry knowledge, the center of rotation may be calculated after eliminating the translation motion out of the 6-DOF motion. The motion equations are educed based on the motion model and the closed-form solutions are figured out. To heighten the motion estimation algorithm＇s robust, RANSAC algorithm is applied to delete the outliers. Simulation and real experiments are conducted and the experiment results are analyzed. The results prove the motion model＇s correction and algorithm＇s validity.

Robot stereo vision calibration method with genetic algorithm and particle swarm optimization

Accurate stereo vision calibration is a preliminary step towards high-precision visual posi- tioning of robot. Combining with the characteristics of genetic algorithm （GA） and particle swarm optimization （PSO）, a three-stage calibration method based on hybrid intelligent optimization is pro- posed for nonlinear camera models in this paper. The motivation is to improve the accuracy of the calibration process. In this approach, the stereo vision calibration is considered as an optimization problem that can be solved by the GA and PSO. The initial linear values can be obtained in the frost stage. Then in the second stage, two cameras＇ parameters are optimized separately. Finally, the in- tegrated optimized calibration of two models is obtained in the third stage. Direct linear transforma- tion （DLT）, GA and PSO are individually used in three stages. It is shown that the results of every stage can correctly find near-optimal solution and it can be used to initialize the next stage. Simula- tion analysis and actual experimental results indicate that this calibration method works more accu- rate and robust in noisy environment compared with traditional calibration methods. The proposed method can fulfill the requirements of robot sophisticated visual operation.

Control of a 6DOF Mobile Manipulator with Object Detection and Tracking Using Stereo Vision

A supportive mobile robot for assisting the elderly is an emerging requirement mainly in countries like Japan where population ageing become relevant in near future.Falls related injuries are considered as a critical issue when taking into account the physical health of older people.A personal assistive robot with the capability of picking up and carrying objects for long/short distances can be used to overcome or lessen this problem.Here,we design and introduce a 3 D dynamic simulation of such an assistive robot to perform pick and place of objects through visual recognition.The robot consists of two major components;a robotic arm or manipulator to do the pick and place,and an omnidirectional wheeled robotic platform to support mobility.Both components are designed and operated according to their kinematics and dynamics and the controllers are integrated for the combined performance.The objective was to improve the accuracy of the robot at a considerably high speed.Designed mobile manipulator has been successfully tested and simulated with a stereo vision system to perform object recognition and tracking in a virtual environment resembling aroom of an elderly care.The tracking accuracy of the mobile manipulator at an average speed of 0.5 m/s is 90%and is well suited for the proposed application.

Development of a Stereo Vision-based Pick and Place System for Robotic Manipulators

Conventional robotic manipulators consist of touch and vision sensors in order to pick and place differently shaped objects.Due to the technology development and degrading sensors over a long period,the stereo vision technique has become a promising alternative.In this study,a low-cost stereo vision-based system,and a gripper to be placed at the end of the robot arm(Fanuc M10 iA/12)are developed for position and orientation estimation of robotic manipulators to pick and place different shaped objects.The stereo vision system developed in this research is used to estimate the position(X,Y,Z),orientation(P_(y))of the Center of Volume of four standard objects(cube,cuboid,cylinder,and sphere)whereas the robot arm with the gripper is used to mechanically pick and place the objects.The stereo vision system is placed on the movable robot arm,and it consists of two cameras to capture two 2D views of a stationary object to derive 3D depth information in 3D space.Moreover,a graphical user interface is developed to train a linear regression model,live predict the coordinates of the objects,and check the accuracy of the predicted data.The graphical user interface can also send predicted coordinates and angles to the gripper and the robot arm.The project is facilitated with python programming language modules and image processing techniques.Identification of the stationary object and estimation of its coordinates is done using image processing techniques.The final product can be identified as a device that converts conventional robot arms without an image processing vision system into a highly precise and accurate robot arm with an image processing vision system.Experimental studies are performed to test the efficiency and effectiveness of used techniques and the gripper prototype.Necessary actions are taken to minimize the errors in position and orientation estimation.In addition,as a future implementation,an embedded system will be developed with a user-friendly software interface to install the vision system into the Fanuc M10 iA/12 robot arm and will upgrade the system to a device that can be implemented with any kind of customized robot arms available in the industry.

Research on Extraction of Bottom of Shoe Pattern Based on Binocular Stereo Vision

In order to quickly and efficiently get the information of the bottom of the shoe pattern and spraying trajectory, the paper proposes a method based on binocular stereo vision. After acquiring target image, edge detection based on the canny algorithm, the paper begins stereo matching based on area and characteristics of algorithm. To eliminate false matching points, the paper uses the principle of polar geometry in computer vision. For the purpose of gaining the 3D point cloud of spraying curve, the paper adopts the principle of binocular stereo vision 3D measurement, and then carries on cubic spline curve fitting. By HALCON image processing software programming, it proves the feasibility and effectiveness of the method

Fast and Robust Stereo Vision Algorithm for Obstacle Detection

Binocular computer vision is based on bionics, after the calibration through the camera head by double-exposure image synchronization, access to the calculation of two-dimensional image pixels of the three-dimensional depth information. In this paper, a fast and robust stereo vision algorithm is described to perform in-vehicle obstacles detection and characterization. The stereo algorithm which provides a suitable representation of the geometric content of the road scene is described, and an in-vehicle embedded system is presented. We present the way in which the algorithm is used, and then report experiments on real situations which show that our solution is accurate, reliable and efficient. In particular, both processes are fast, generic, robust to noise and bad conditions, and work even with partial occlusion.

Convergence of Stereo Vision-Based Multimodal YOLOs for FasterDetection of Potholes

Road potholes can cause serious social issues,such as unexpected damages to vehicles and traffic accidents.For efficient road management,technologies that quickly find potholes are required,and thus researches on such technologies have been conducted actively.The three-dimensional(3D)reconstruction method has relatively high accuracy and can be used in practice but it has limited application owing to its long data processing time and high sensor maintenance cost.The two-dimensional(2D)vision method has the advantage of inexpensive and easy application of sensor.Recently,although the 2D vision method using the convolutional neural network(CNN)has shown improved pothole detection performance and adaptability,large amount of data is required to sufficiently train the CNN.Therefore,we propose a method to improve the learning performance of CNN-based object detection model by artificially generating synthetic data similar to a pothole and enhancing the learning data.Additionally,to make the defective areas appear more contrasting,the transformed disparity map(TDM)was calculated using stereo-vision cameras,and the detection performance of the model was further improved through the late fusion with RGB(Red,Green,Blue)images.Consequently,through the convergence of multimodal You Only Look Once(YOLO)frameworks trained by RGB images and TDMs respectively,the detection performance was enhanced by 10.7%compared with that when using only RGB.Further,the superiority of the proposed method was confirmed by showing that the data processing speed was two times faster than the existing 3D reconstruction method.

Structured scene modeling using micro stereo vision system with large field of view

This paper presents a method for structured scene modeling using micro stereo vision system with large field of view. The proposed algorithm includes edge detection with Canny detector, line fitting with principle axis based approach, finding corresponding lines using feature based matching method, and 3D line depth computation.

Deforming analysis of sheet metal based on stereo vision and coordinate grid

A new approach based on stereo vision technology is introduced to analyzesheet metal deformation. By measuring the deformed circle grids that are printed on the sheetsurface before forming, the strain distribution of the workpiece is obtained. The measurement andanalysis results can be used to verify numerical simulation results and guide production. To getgood accuracy, some new techniques are employed: camera calibration based on genetic algorithm,feature abstraction based on self-adaptive technology, image matching based on structure feature andcamera modeling pre-constrains, and parameter calculation based on curve and surface optimization.The experimental values show that the approach proposed is rational and practical, which can providebetter measurement accuracy with less time than the conventional method.

Automatic three-dimensional reconstruction based on four-view stereo vision using checkerboard pattern

An automatic three-dimensional(3D) reconstruction method based on four-view stereo vision using checkerboard pattern is presented. Mismatches easily exist in traditional binocular stereo matching due to the repeatable or similar features of binocular images. In order to reduce the probability of mismatching and improve the measure precision, a four-camera measurement system which can add extra matching constraints and offer multiple measurements is applied in this work. Moreover, a series of different checkerboard patterns are projected onto the object to obtain dense feature points and remove mismatched points. Finally, the 3D model is generated by performing Delaunay triangulation and texture mapping on the point cloud obtained by four-view matching. This method was tested on the 3D reconstruction of a terracotta soldier sculpture and the Buddhas in the Mogao Grottoes. Their point clouds without mismatched points were obtained and less processing time was consumed in most cases relative to binocular matching. These good reconstructed models show the effectiveness of the method.

Cast shadow detection for moving objects based on binocular stereo vision

A novel cast shadow detection approach was proposed.A stereo vision system was used to capture images instead of traditional single camera.It was based on an assumption that cast shadows were on a special plane.The image obtained from one camera was inversely projected to the plane and then transformed to the view from another camera.The points on the plane shared the same position between original image and the transformed image.As a result,the cast shadows can be detected.In order to improve the efficiency of cast shadow detection and decrease computational complexity,the obvious object areas in CIELAB color space were removed and the potential shadow areas were obtained.Experimental results demonstrate that the proposed approach can detect cast shadows accurately even under various illuminations.

Research on stereo vision path-planning algorithms for mobile robots autonomous navigation

Using stereo vision for autonomous mobile robot path-planning is a hot technology.The environment mapping and path-planning algorithms were introduced,and they were applied in the autonomous mobile robot experiment platform.Through experiments in the robot platform,the effectiveness of these algorithms was verified.

DFPC-SLAM:A Dynamic Feature Point Constraints-Based SLAM Using Stereo Vision for Dynamic Environment

Visual SLAM methods usually presuppose that the scene is static, so the SLAM algorithm formobile robots in dynamic scenes often results in a signicant decrease in accuracy due to thein°uence of dynamic objects. In this paper, feature points are divided into dynamic and staticfrom semantic information and multi-view geometry information, and then static region featurepoints are added to the pose-optimization, and static scene maps are established for dynamicscenes. Finally, experiments are conducted in dynamic scenes using the KITTI dataset, and theresults show that the proposed algorithm has higher accuracy in highly dynamic scenes comparedto the visual SLAM baseline.

基于Stereo Camera-YOLOv5自然环境下百香果检测与定位模型

针对百香果采摘机器人在自然环境中作业时受复杂光线及遮挡影响,难以快速精确地检测及定位成熟百香果的问题,提出一种基于Stereo Camera-YOLOv5自然环境下成熟百香果检测及定位模型。针对自然环境下光线以及遮挡的影响,通过MSRCP算法、随机遮挡、数据增扩等图像处理算法对原始数据集进行优化。将优化的数据集输入到YOLOv5网络中训练出最优模型,在检测代码中嵌入双目立体视觉算法。该模型对自然环境下百香果进行检测及成熟度判断,将判断为成熟的百香果进行图像处理,并提取到中心点二维坐标。通过立体匹配及视差计算得到中心点的三维坐标。田间试验结果表明,该模型的目标检测准确率为97.8%,总体准确率为90.2%,平均运行时间为4.85 s。该系统鲁棒性强、实时性好,能够更好地实现自然环境下成熟百香果的检测及定位,为百香果采摘机器人后续工作奠定基础。

Accuracy study of a binocular-stereo-vision-based navigation robot for minimally invasive interventional procedures

BACKGROUND Medical robot is a promising surgical tool,but no specific one has been designed for interventional treatment of chronic pain.We developed a computed tomography-image based navigation robot using a new registration method with binocular vision.This kind of robot is appropriate for minimal invasive interventional procedures and easy to operate.The feasibility,accuracy and stability of this new robot need to be tested.AIM To assess quantitatively the feasibility,accuracy and stability of the binocularstereo-vision-based navigation robot for minimally invasive interventional procedures.METHODS A box model was designed for assessing the accuracy for targets at different distances.Nine(three sets)lead spheres were embedded in the model as puncture goals.The entry-to-target distances were set 50 mm(short-distance),100 mm(medium-distance)and 150 mm(long-distance).Puncture procedure was repeated three times for each goal.The Euclidian error of each puncture was calculated and statistically analyzed.Three head phantoms were used to explore the clinical feasibility and stability.Three independent operators conducted foramen ovale placement on head phantoms(both sides)by freehand or under the guidance of robot(18 punctures with each method).The operation time,adjustment time and one-time success rate were recorded,and the two guidancemethods were compared.RESULTS On the box model,the mean puncture errors of navigation robot were 1.7±0.9 mm for the short-distance target,2.4±1.0 mm for the moderate target and 4.4±1.4 mm for the long-distance target.On the head phantom,no obvious differences in operation time and adjustment time were found among the three performers(P>0.05).The median adjustment time was significantly less under the guidance of the robot than under free hand.The one-time success rate was significantly higher with the robot(P<0.05).There was no obvious difference in operation time between the two methods(P>0.05).CONCLUSION In the laboratory environment,accuracy of binocular-stereo-vision-based navigation robot is acceptable for target at 100 mm depth or less.Compared with freehand,foramen ovale placement accuracy can be improved with robot guidance.

In-vehicle stereo vision system for identification of traffic conflicts between bus and pedestrian

The traffic conflict technique （TCT） was developed as ＂surrogate measure of road safety＂ to identify near-crash events by using measures of the spatial and temporal proximity of road users. Traditionally applications of TCT focus on a specific site by the way of manually or automated supervision. Nowadays the development of in-vehicle （IV） technologies pro- vides new opportunities for monitoring driver behavior and interaction with other road users directly into the traffic stream. In the paper a stereo vision and GPS system for traffic conflict investigation is presented for detecting conflicts between vehicle and pedestrian. The system is able to acquire geo-referenced sequences of stereo frames that are used to provide real time information related to conflict occurrence and severity. As case study, an urban bus was equipped with a prototype of the system and a trial in the city of Catania （Italy） was carried out analyzing conflicts with pedestrian crossing in front of the bus. Experimental results pointed out the potentialities of the system for collection of data that can be used to get suitable traffic conflict measures. Specifically, a risk index of the conflict between pedestrians and vehicles is proposed to classify collision probability and severity using data collected by the system. This information may be used to develop in-vehicle warning systems and urban network risk assessment.

A novel virtual four-ocular stereo vision system based on single camera for measuring insect motion parameters

A novel virtual four-ocular stereo measurement system based on single high speed camera is proposed for measuring double beating wings of a high speed flapping insect. The principle of virtual monocular system consisting of a few planar mirrors and a single high speed camera is introduced. The stereo vision measurement principle based on optic triangulation is explained. The wing kinematics parameters are measured. Results show that this virtual stereo system not only decreases system cost extremely but also is effective to insect motion measurement.

Bubble behavior characteristics based on virtual binocular stereo vision

The three-dimensional(3D) behavior characteristics of bubble rising in gas-liquid two-phase flow are of great importance to study bubbly flow mechanism and guide engineering practice. Based on the dual-perspective imaging of virtual binocular stereo vision, the 3D behavior characteristics of bubbles in gas-liquid two-phase flow are studied in detail, which effectively increases the projection information of bubbles to acquire more accurate behavior features. In this paper, the variations of bubble equivalent diameter, volume, velocity and trajectory in the rising process are estimated, and the factors affecting bubble behavior characteristics are analyzed. It is shown that the method is real-time and valid, the equivalent diameter of the rising bubble in the stagnant water is periodically changed, and the crests and troughs in the equivalent diameter curve appear alternately. The bubble behavior characteristics as well as the spiral amplitude are affected by the orifice diameter and the gas volume flow.

Large-scale 3D Semantic Mapping Using Stereo Vision

In recent years, there have been a lot of interests in incorporating semantics into simultaneous localization and mapping （SLAM） systems. This paper presents an approach to generate an outdoor large-scale 3D dense semantic map based on binocular stereo vision. The inputs to system are stereo color images from a moving vehicle. First, dense 3D space around the vehicle is constructed, and tile motion of camera is estimated by visual odometry. Meanwhile, semantic segmentation is performed through the deep learning technology online, and the semantic labels are also used to verify tim feature matching in visual odometry. These three processes calculate the motion, depth and semantic label of every pixel in the input views. Then, a voxel conditional random field （CRF） inference is introduced to fuse semantic labels to voxel. After that, we present a method to remove the moving objects by incorporating the semantic labels, which improves the motion segmentation accuracy. The last is to generate tile dense 3D semantic map of an urban environment from arbitrary long image sequence. We evaluate our approach on KITTI vision benchmark, and the results show that the proposed method is effective.