Disparity estimation for multi-scale multi-sensor fusion

The perception module of advanced driver assistance systems plays a vital role.Perception schemes often use a single sensor for data processing and environmental perception or adopt the information processing results of various sensors for the fusion of the detection layer.This paper proposes a multi-scale and multi-sensor data fusion strategy in the front end of perception and accomplishes a multi-sensor function disparity map generation scheme.A binocular stereo vision sensor composed of two cameras and a light deterction and ranging(LiDAR)sensor is used to jointly perceive the environment,and a multi-scale fusion scheme is employed to improve the accuracy of the disparity map.This solution not only has the advantages of dense perception of binocular stereo vision sensors but also considers the perception accuracy of LiDAR sensors.Experiments demonstrate that the multi-scale multi-sensor scheme proposed in this paper significantly improves disparity map estimation.

Generation of triangle mesh surface from the disparity map for telerobotic welding

3D reconstruction of environment and weld workpiece can provide geometrical model for telerobotic welding and improve its intelligence. A novel framework of spacetime stereo is employed to overcome the lack of texture of the weld workpiece and obtain subpixel disparity map of the scene. Anisotropic diffusion is adopted to smooth the original subpixel disparity map in order to reduce the noise while preserving the depth discontinuity. A simple algorithm of generation triangle mesh surface from the disparity map of the spucetime stereo is presented. The experimental results of real weld scenes are shown.

A Robust Algorithm Based on Object Contours and Order Matching for Disparity Map Post-Processing

Based on the feature of stereo images' content and the property of natural objects, we redefine the general order matching constraint with object contour restriction. According to the modified order matching constraint, we propose a robust algorithm for disparity map post processing. Verified by computer simulations using synthetic stereo images with given disparities, our new algorithm proves to be not only efficient in disparity error detection and correction, but also very robust, which can resolve the severe problem in the algorithm proposed in Ref. that if there are large differences among the depths of objects in a scene, the algorithm will make mistakes during the process of disparity error detection and correction.

Depth Map Prediction of Occluded Objects Using Structure Tensor with Gain Regularization

The creation of the 3D rendering model involves the prediction of an accurate depth map for the input images.A proposed approach of a modified semi-global block matching algorithm with variable window size and the gradient assessment of objects predicts the depth map.3D modeling and view synthesis algorithms could effectively handle the obtained disparity maps.This work uses the consistency check method to find an accurate depth map for identifying occluded pixels.The prediction of the disparity map by semi-global block matching has used the benchmark dataset of Middlebury stereo for evaluation.The improved depth map quality within a reasonable process-ing time outperforms the other existing depth map prediction algorithms.The experimental results have shown that the proposed depth map predictioncould identify the inter-object boundaryeven with the presence ofocclusion with less detection error and runtime.We observed that the Middlebury stereo dataset has very few images with occluded objects,which made the attainment of gain cumbersome.Considering this gain,we have created our dataset with occlu-sion using the structured lighting technique.The proposed regularization term as an optimization process in the graph cut algorithm handles occlusion for different smoothing coefficients.The experimented results demonstrated that our dataset had outperformed the Tsukuba dataset regarding the percentage of occluded pixels.

An improved seed growth method for accurate stereo matching in disparity space

Matching is a classical problem in stereo vision. To solve the matching problem that components cannot continue growing on the occlusions region and repetitive patterns, an improved seed growth method is proposed. The method obtains a set of interesting points defined as initial seeds from a rectified image. Through global optimization the seeds and their neighbors can be selected in- to a match table. Finally the components grow with the matching points and create a semi-dense map under the maximum similar subset according to the principle of the unique constraint. Experimental results show that the proposed method in the grown process can rectify some errors in matching. The semi-dense map has a good performance in the occlusions region and repetitive patterns. This algorithm is faster and more accurate than the traditional seed growing method.

A novel algorithm for distance measurement using stereo camera

Distance estimation can be achieved by using active sensors or with the help of passive sensors such as cameras.The stereo vision system is generally composed of two cameras to mimic the human binocular vision.In this paper,a Python-based algorithm is pro-posed to find the parameters of each camera,rectify the images,create the disparity maps and finally use these maps for distance measurements.Experiments using real-time im-ages,which were captured from our stereo vision system,of different obstacles posi-tioned at multiple distances(60-200 cm)prove the effectiveness of the proposed program and show that the calculated distance to the obstacle is relatively accurate.The accuracy of distance measurement is up to 99.83%.The processing time needed to calculate the distance between the obstacle and the cameras is less than 0.355 s.

Convergence of Stereo Vision-Based Multimodal YOLOs for FasterDetection of Potholes

Road potholes can cause serious social issues,such as unexpected damages to vehicles and traffic accidents.For efficient road management,technologies that quickly find potholes are required,and thus researches on such technologies have been conducted actively.The three-dimensional(3D)reconstruction method has relatively high accuracy and can be used in practice but it has limited application owing to its long data processing time and high sensor maintenance cost.The two-dimensional(2D)vision method has the advantage of inexpensive and easy application of sensor.Recently,although the 2D vision method using the convolutional neural network(CNN)has shown improved pothole detection performance and adaptability,large amount of data is required to sufficiently train the CNN.Therefore,we propose a method to improve the learning performance of CNN-based object detection model by artificially generating synthetic data similar to a pothole and enhancing the learning data.Additionally,to make the defective areas appear more contrasting,the transformed disparity map(TDM)was calculated using stereo-vision cameras,and the detection performance of the model was further improved through the late fusion with RGB(Red,Green,Blue)images.Consequently,through the convergence of multimodal You Only Look Once(YOLO)frameworks trained by RGB images and TDMs respectively,the detection performance was enhanced by 10.7%compared with that when using only RGB.Further,the superiority of the proposed method was confirmed by showing that the data processing speed was two times faster than the existing 3D reconstruction method.