Hawk‐eye‐inspired perception algorithm of stereo vision for obtaining orchard 3D point cloud navigation map

The binocular stereo vision is the lowest cost sensor for obtaining 3D information.Considering the weakness of long‐distance measurement and stability,the improvement of accuracy and stability of stereo vision is urgently required for application of precision agriculture.To address the challenges of stereo vision long‐distance measurement and stable perception without hardware upgrade,inspired by hawk eyes,higher resolution perception and the adaptive HDR(High Dynamic Range)were introduced in this paper.Simulating the function from physiological structure of‘deep fovea’and‘shallow fovea’of hawk eye,the higher resolution reconstruction method in this paper was aimed at ac-curacy improving.Inspired by adjustment of pupils,the adaptive HDR method was proposed for high dynamic range optimisation and stable perception.In various light conditions,compared with default stereo vision,the accuracy of proposed algorithm was improved by 28.0%evaluated by error ratio,and the stability was improved by 26.56%by disparity accuracy.For fixed distance measurement,the maximum improvement was 78.6%by standard deviation.Based on the hawk‐eye‐inspired perception algorithm,the point cloud of orchard was improved both in quality and quantity.The hawk‐eye‐inspired perception algorithm contributed great advance in binocular 3D point cloud recon-struction in orchard navigation map.

Wheel center detection based on stereo vision

As the location of the wheel center is the key to accurately measuring the wheelbase, the wheelbase difference and the wheel static radius, a high-precision wheel center detection method based on stereo vision is proposed. First, according to the prior information, the contour of the wheel hub is extracted and fitted as an ellipse curve, and the ellipse fitting equation can be obtained. Then, a new un-tangent constraint is adopted to improve the ellipse matching precision. Finally, the 3D coordinates of the wheel center can be reconstructed by the spatial circle projection algorithm with low time complexity and high measurement accuracy. Simulation experiments verify that compared with the ellipse center reconstruction algorithm and the planar constraint optimization algorithm, the proposed method can acquire the 3D coordinates of the spatial circle more exactly. Furthermore, the measurements of the wheelbase, the wheelbase difference and the wheel static radius for three types of vehicles demonstrate the effectiveness of the proposed method for wheel center detection.

Measurement method for cluster warhead's dispersion area based on binocular stereo vision technique

Cluster warhead has become the main ammunition of gun,rocket projectile and missile and it has been widely equipped in almost every country.More and more attention is paid to the damage effect of cluster warhead.The size of the dispersion area of cluster warhead is the main standard by which the damage effect of cluster warhead is estimated.The practical method of measuring the dispersion area was developed based on binocular stereo vision measurement theory.The calibration principle of the binocular stereo vision cameras was studied.The matching algorithm that relies on the gradient fields of the neighborhood of a pixel has been used to obtain the spatial information of matched points by acquiring apair of corresponding points in the left and right images of binocular cameras.The 3Dpositions of the flying path of cluster warhead were calculated.The umbrella that is similar to the dispersion track of static explosive cluster warhead was applied in the experiment to get the projection area of the umbrella on the ground.Experiment results verify the feasibility of the proposed method.

6-DOF MOTION AND CENTER OF ROTATION ESTIMATION BASED ON STEREO VISION

A new motion model and estimation algorithm is proposed to compute the general rigid motion object＇s 6-DOF motion parameters and center of rotation based on stereo vision. The object＇s 6-DOF motion model is designed from the rigid object＇s motion character under the two defined reference frames. According to the rigid object＇s motion model and motion dynamics knowledge, the corresponding motion algorithm to compute the 6-DOF motion parameters is worked out. By the rigid object pure rotation motion model and space sphere geometry knowledge, the center of rotation may be calculated after eliminating the translation motion out of the 6-DOF motion. The motion equations are educed based on the motion model and the closed-form solutions are figured out. To heighten the motion estimation algorithm＇s robust, RANSAC algorithm is applied to delete the outliers. Simulation and real experiments are conducted and the experiment results are analyzed. The results prove the motion model＇s correction and algorithm＇s validity.

Measuring Propeller Blade Width Using Binocular Stereo Vision

Propeller blade width measurement has been extensively studied in the past using direct and indirect methods, and it plays a great role in determining the quality of the finished products. It has surveyed that previous techniques are usually time-consuming and erroneous due to a large number of points to be processed in blade width measurement. This paper proposes a new method of measuring blade width using two images acquired from different viewpoints of the same blade. And a new feature points matching approach for propeller blade image is proposed in stereo vision measurement. Based on these, pixel coordinates of contour points of the blade in two images are extracted and converted to real world coordinates by image algorithm and binocular stereo machine vision theory. Then, from the real world coordinates, the blade width at any position can be determined by simple geometrical method.

Cast shadow detection for moving objects based on binocular stereo vision

A novel cast shadow detection approach was proposed.A stereo vision system was used to capture images instead of traditional single camera.It was based on an assumption that cast shadows were on a special plane.The image obtained from one camera was inversely projected to the plane and then transformed to the view from another camera.The points on the plane shared the same position between original image and the transformed image.As a result,the cast shadows can be detected.In order to improve the efficiency of cast shadow detection and decrease computational complexity,the obvious object areas in CIELAB color space were removed and the potential shadow areas were obtained.Experimental results demonstrate that the proposed approach can detect cast shadows accurately even under various illuminations.

Robot stereo vision calibration method with genetic algorithm and particle swarm optimization

Accurate stereo vision calibration is a preliminary step towards high-precision visual posi- tioning of robot. Combining with the characteristics of genetic algorithm （GA） and particle swarm optimization （PSO）, a three-stage calibration method based on hybrid intelligent optimization is pro- posed for nonlinear camera models in this paper. The motivation is to improve the accuracy of the calibration process. In this approach, the stereo vision calibration is considered as an optimization problem that can be solved by the GA and PSO. The initial linear values can be obtained in the frost stage. Then in the second stage, two cameras＇ parameters are optimized separately. Finally, the in- tegrated optimized calibration of two models is obtained in the third stage. Direct linear transforma- tion （DLT）, GA and PSO are individually used in three stages. It is shown that the results of every stage can correctly find near-optimal solution and it can be used to initialize the next stage. Simula- tion analysis and actual experimental results indicate that this calibration method works more accu- rate and robust in noisy environment compared with traditional calibration methods. The proposed method can fulfill the requirements of robot sophisticated visual operation.

Control of a 6DOF Mobile Manipulator with Object Detection and Tracking Using Stereo Vision

A supportive mobile robot for assisting the elderly is an emerging requirement mainly in countries like Japan where population ageing become relevant in near future.Falls related injuries are considered as a critical issue when taking into account the physical health of older people.A personal assistive robot with the capability of picking up and carrying objects for long/short distances can be used to overcome or lessen this problem.Here,we design and introduce a 3 D dynamic simulation of such an assistive robot to perform pick and place of objects through visual recognition.The robot consists of two major components;a robotic arm or manipulator to do the pick and place,and an omnidirectional wheeled robotic platform to support mobility.Both components are designed and operated according to their kinematics and dynamics and the controllers are integrated for the combined performance.The objective was to improve the accuracy of the robot at a considerably high speed.Designed mobile manipulator has been successfully tested and simulated with a stereo vision system to perform object recognition and tracking in a virtual environment resembling aroom of an elderly care.The tracking accuracy of the mobile manipulator at an average speed of 0.5 m/s is 90%and is well suited for the proposed application.

Development of a Stereo Vision-based Pick and Place System for Robotic Manipulators

Conventional robotic manipulators consist of touch and vision sensors in order to pick and place differently shaped objects.Due to the technology development and degrading sensors over a long period,the stereo vision technique has become a promising alternative.In this study,a low-cost stereo vision-based system,and a gripper to be placed at the end of the robot arm(Fanuc M10 iA/12)are developed for position and orientation estimation of robotic manipulators to pick and place different shaped objects.The stereo vision system developed in this research is used to estimate the position(X,Y,Z),orientation(P_(y))of the Center of Volume of four standard objects(cube,cuboid,cylinder,and sphere)whereas the robot arm with the gripper is used to mechanically pick and place the objects.The stereo vision system is placed on the movable robot arm,and it consists of two cameras to capture two 2D views of a stationary object to derive 3D depth information in 3D space.Moreover,a graphical user interface is developed to train a linear regression model,live predict the coordinates of the objects,and check the accuracy of the predicted data.The graphical user interface can also send predicted coordinates and angles to the gripper and the robot arm.The project is facilitated with python programming language modules and image processing techniques.Identification of the stationary object and estimation of its coordinates is done using image processing techniques.The final product can be identified as a device that converts conventional robot arms without an image processing vision system into a highly precise and accurate robot arm with an image processing vision system.Experimental studies are performed to test the efficiency and effectiveness of used techniques and the gripper prototype.Necessary actions are taken to minimize the errors in position and orientation estimation.In addition,as a future implementation,an embedded system will be developed with a user-friendly software interface to install the vision system into the Fanuc M10 iA/12 robot arm and will upgrade the system to a device that can be implemented with any kind of customized robot arms available in the industry.

Research on Extraction of Bottom of Shoe Pattern Based on Binocular Stereo Vision

In order to quickly and efficiently get the information of the bottom of the shoe pattern and spraying trajectory, the paper proposes a method based on binocular stereo vision. After acquiring target image, edge detection based on the canny algorithm, the paper begins stereo matching based on area and characteristics of algorithm. To eliminate false matching points, the paper uses the principle of polar geometry in computer vision. For the purpose of gaining the 3D point cloud of spraying curve, the paper adopts the principle of binocular stereo vision 3D measurement, and then carries on cubic spline curve fitting. By HALCON image processing software programming, it proves the feasibility and effectiveness of the method

Fast and Robust Stereo Vision Algorithm for Obstacle Detection

Binocular computer vision is based on bionics, after the calibration through the camera head by double-exposure image synchronization, access to the calculation of two-dimensional image pixels of the three-dimensional depth information. In this paper, a fast and robust stereo vision algorithm is described to perform in-vehicle obstacles detection and characterization. The stereo algorithm which provides a suitable representation of the geometric content of the road scene is described, and an in-vehicle embedded system is presented. We present the way in which the algorithm is used, and then report experiments on real situations which show that our solution is accurate, reliable and efficient. In particular, both processes are fast, generic, robust to noise and bad conditions, and work even with partial occlusion.

Convergence of Stereo Vision-Based Multimodal YOLOs for FasterDetection of Potholes

Road potholes can cause serious social issues,such as unexpected damages to vehicles and traffic accidents.For efficient road management,technologies that quickly find potholes are required,and thus researches on such technologies have been conducted actively.The three-dimensional(3D)reconstruction method has relatively high accuracy and can be used in practice but it has limited application owing to its long data processing time and high sensor maintenance cost.The two-dimensional(2D)vision method has the advantage of inexpensive and easy application of sensor.Recently,although the 2D vision method using the convolutional neural network(CNN)has shown improved pothole detection performance and adaptability,large amount of data is required to sufficiently train the CNN.Therefore,we propose a method to improve the learning performance of CNN-based object detection model by artificially generating synthetic data similar to a pothole and enhancing the learning data.Additionally,to make the defective areas appear more contrasting,the transformed disparity map(TDM)was calculated using stereo-vision cameras,and the detection performance of the model was further improved through the late fusion with RGB(Red,Green,Blue)images.Consequently,through the convergence of multimodal You Only Look Once(YOLO)frameworks trained by RGB images and TDMs respectively,the detection performance was enhanced by 10.7%compared with that when using only RGB.Further,the superiority of the proposed method was confirmed by showing that the data processing speed was two times faster than the existing 3D reconstruction method.

Structured scene modeling using micro stereo vision system with large field of view

This paper presents a method for structured scene modeling using micro stereo vision system with large field of view. The proposed algorithm includes edge detection with Canny detector, line fitting with principle axis based approach, finding corresponding lines using feature based matching method, and 3D line depth computation.

Deforming analysis of sheet metal based on stereo vision and coordinate grid

A new approach based on stereo vision technology is introduced to analyzesheet metal deformation. By measuring the deformed circle grids that are printed on the sheetsurface before forming, the strain distribution of the workpiece is obtained. The measurement andanalysis results can be used to verify numerical simulation results and guide production. To getgood accuracy, some new techniques are employed: camera calibration based on genetic algorithm,feature abstraction based on self-adaptive technology, image matching based on structure feature andcamera modeling pre-constrains, and parameter calculation based on curve and surface optimization.The experimental values show that the approach proposed is rational and practical, which can providebetter measurement accuracy with less time than the conventional method.

Automatic three-dimensional reconstruction based on four-view stereo vision using checkerboard pattern

An automatic three-dimensional(3D) reconstruction method based on four-view stereo vision using checkerboard pattern is presented. Mismatches easily exist in traditional binocular stereo matching due to the repeatable or similar features of binocular images. In order to reduce the probability of mismatching and improve the measure precision, a four-camera measurement system which can add extra matching constraints and offer multiple measurements is applied in this work. Moreover, a series of different checkerboard patterns are projected onto the object to obtain dense feature points and remove mismatched points. Finally, the 3D model is generated by performing Delaunay triangulation and texture mapping on the point cloud obtained by four-view matching. This method was tested on the 3D reconstruction of a terracotta soldier sculpture and the Buddhas in the Mogao Grottoes. Their point clouds without mismatched points were obtained and less processing time was consumed in most cases relative to binocular matching. These good reconstructed models show the effectiveness of the method.

Research on stereo vision path-planning algorithms for mobile robots autonomous navigation

Using stereo vision for autonomous mobile robot path-planning is a hot technology.The environment mapping and path-planning algorithms were introduced,and they were applied in the autonomous mobile robot experiment platform.Through experiments in the robot platform,the effectiveness of these algorithms was verified.

FlockSeer:A portable stereo vision observer for bird flocking

Bird flocking is a paradigmatic case of self‐organised collective behaviours in biology.Stereo camera systems are employed to observe flocks of starlings,jackdaws,and chimney swifts,mainly on a spot‐fixed basis.A portable non‐fixed stereo vision‐based flocking observation system,namely FlockSeer,is developed by the authors for observing more species of bird flocks within field scenarios.The portable flocking observer,FlockSeer,responds to the challenges in extrinsic calibration,camera synchronisation and field movability compared to existing spot‐fixed observing systems.A measurement and sensor fusion approach is utilised for rapid calibration,and a light‐based synchronisation approach is used to simplify hardware configuration.FlockSeer has been implemented and tested across six cities in three provinces and has accomplished diverse flock‐tracking tasks,accumulating behavioural data of four species,including egrets,with up to 300 resolvable trajectories.The authors reconstructed the trajectories of a flock of egrets under disturbed conditions to verify the practicality and reliability.In addition,we analysed the accuracy of identifying nearest neighbours,and then examined the similarity between the trajectories and the Couzin model.Experimental results demonstrate that the developed flocking observing system is highly portable,more convenient and swift to deploy in wetland‐like or coast‐like fields.Its observation process is reliable and practical and can effectively support the study of understanding and modelling of bird flocking behaviours.

DFPC-SLAM:A Dynamic Feature Point Constraints-Based SLAM Using Stereo Vision for Dynamic Environment

Visual SLAM methods usually presuppose that the scene is static, so the SLAM algorithm formobile robots in dynamic scenes often results in a signicant decrease in accuracy due to thein°uence of dynamic objects. In this paper, feature points are divided into dynamic and staticfrom semantic information and multi-view geometry information, and then static region featurepoints are added to the pose-optimization, and static scene maps are established for dynamicscenes. Finally, experiments are conducted in dynamic scenes using the KITTI dataset, and theresults show that the proposed algorithm has higher accuracy in highly dynamic scenes comparedto the visual SLAM baseline.

Warhead fragments motion trajectories tracking and spatio-temporal distribution reconstruction method based on high-speed stereo photography

High speed photography technique is potentially the most effective way to measure the motion parameter of warhead fragment benefiting from its advantages of high accuracy,high resolution and high efficiency.However,it faces challenge in dense objects tracking and 3D trajectories reconstruction due to the characteristics of small size and dense distribution of fragment swarm.To address these challenges,this work presents a warhead fragments motion trajectories tracking and spatio-temporal distribution reconstruction method based on high-speed stereo photography.Firstly,background difference algorithm is utilized to extract the center and area of each fragment in the image sequence.Subsequently,a multi-object tracking(MOT)algorithm using Kalman filtering and Hungarian optimal assignment is developed to realize real-time and robust trajectories tracking of fragment swarm.To reconstruct 3D motion trajectories,a global stereo trajectories matching strategy is presented,which takes advantages of epipolar constraint and continuity constraint to correctly retrieve stereo correspondence followed by 3D trajectories refinement using polynomial fitting.Finally,the simulation and experimental results demonstrate that the proposed method can accurately track the motion trajectories and reconstruct the spatio-temporal distribution of 1.0×10^(3)fragments in a field of view(FOV)of 3.2 m×2.5 m,and the accuracy of the velocity estimation can achieve 98.6%.

基于Stereo Camera-YOLOv5自然环境下百香果检测与定位模型

针对百香果采摘机器人在自然环境中作业时受复杂光线及遮挡影响,难以快速精确地检测及定位成熟百香果的问题,提出一种基于Stereo Camera-YOLOv5自然环境下成熟百香果检测及定位模型。针对自然环境下光线以及遮挡的影响,通过MSRCP算法、随机遮挡、数据增扩等图像处理算法对原始数据集进行优化。将优化的数据集输入到YOLOv5网络中训练出最优模型,在检测代码中嵌入双目立体视觉算法。该模型对自然环境下百香果进行检测及成熟度判断,将判断为成熟的百香果进行图像处理,并提取到中心点二维坐标。通过立体匹配及视差计算得到中心点的三维坐标。田间试验结果表明,该模型的目标检测准确率为97.8%,总体准确率为90.2%,平均运行时间为4.85 s。该系统鲁棒性强、实时性好,能够更好地实现自然环境下成熟百香果的检测及定位,为百香果采摘机器人后续工作奠定基础。