Seam Tracking and Visual Control for Robotic Arc Welding Based on Structured Light Stereovision

A real-time arc welding robot visual control system based on a local network with a multi-level hierarchy is developed in this paper. It consists of an intelligence and human-machine interface level, a motion planning level, a motion control level and a servo control level. The last three levels form a local real-time open robot controller, which realizes motion planning and motion control of a robot. A camera calibration method based on the relative movement of the end-effector connected to a robot is proposed and a method for tracking weld seam based on the structured light stereovision is provided. Combining the parameters of the cameras and laser plane, three groups of position values in Cartesian space are obtained for each feature point in a stripe projected on the weld seam. The accurate three-dimensional position of the edge points in the weld seam can be calculated from the obtained parameters with an information fusion algorithm. By calculating the weld seam parameter from position and image data, the movement parameters of the robot used for tracking can be determined. A swing welding experiment of type V groove weld is successfully conducted, the results of which show that the system has high resolution seam tracking in real-time, and works stably and efficiently.

Nonlinear optics with structured light

The interest in tailoring light in all its degrees of freedom is steadily gaining traction,driven by the tremendous developments in the toolkit for the creation,control and detection of what is now called structured light.Because the complexity of these optical fields is generally understood in terms of interference,the tools have historically been linear optical elements that create the desired superpositions.For this reason,despite the long and impressive history of nonlinear optics,only recently has the spatial structure of light in nonlinear processes come to the fore.In this review we provide a concise theoretical framework for understanding nonlinear optics in the context of structured light,offering an overview and perspective on the progress made,and the challenges that remain.

Nonlinear effect of the structured light profilometry in the phase-shifting method and error correction

Digital structured light （SL） profilometry is increasingly used in three-dimensional （3D） measurement technology. However, the nonlinearity of the off-the-shelf projectors and cameras seriously reduces the measurement accuracy. In this paper, first, we review the nonlinear effects of the projector-camera system in the phase-shifting structured light depth measurement method. We show that high order harmonic wave components lead to phase error in the phase-shifting method. Then a practical method based on frequency domain filtering is proposed for nonlinear error reduction. By using this method, the nonlinear calibration of the SL system is not required. Moreover, both the nonlinear effects of the projector and the camera can be effectively reduced. The simulations and experiments have verified our nonlinear correction method.

On-line detection system of weld-based rapid prototyping process based on structured light

In order to realize the closed-loop control for rapid prototyping process based on gas metal arc welding, the geometric parameters of weld beads should be detected. In this study, a vision sensor system consisting of a linear laser projector and a CCD camera was designed to collect images of weld beads. Then, an image processing approach which combines with a Gaassian filter and an improved gravity method was used to extract the centerline of a light stripe based on VC ＋＋. Feature points of the centerline were identical directly by means of an image fusion algorithm. Experimental results show that image fusion is an effective approach to measure the width and height of the weld bead with high accuracy. This method can identify beads effectively in multi-pass welding and avoid designing different modes to suit all kinds of shapes.

3D surface reconstruction based on binocular vision using structured light

A 3D surface reconstruction method using a binocular stereo vision technology and a coded structured light,which combines a gray code with phase-shift has been studied.The accuracy of the 3 D surface reconstruction mainly depends on the decoding of gray code views and phase-shift views.In order to find the boundary accurately,gray code patterns and their inverses are projected onto a human eye plaster model.The period dislocation between the gray code views and the phase-shift views in the course of decoding has been analyzed and a new method has been proposed to solve it.The splicing method is based on feature points.The result of the 3D surface reconstruction shows the accuracy and reliability of our method.

Construction for M-arrays and application in structured light

M-arrays are random arrays in which an appropriate sub-window appears only once in the whole array. Coded structured light based on M-arrays is one=shot technique to rapidly acquire 3D information of unknown surfaces by projecting suitable patterns onto a measuring surface. This paper presents a method to construct large size M-arrays based on the piece growing algorithm in which an array is constructed by many pieces through splicing each other. Reconstructing 3D shapes by utilizing the designed pattern based on constructed M-arrays for two objects are given.

SEGMENTATION ALGORITHM BASED ON EDGE-SEARCHING FOR MULTI-LINEAR STRUCTURED LIGHT IMAGES

Aiming at the problem that the existence of disturbances on the edges of light-stripe makes the segmentation of the light-stripes images difficult, a new segmentation algorithm based on edge-searching is presented. It firstly calculates every edge pixel＇s horizontal coordinate grads to produce the corresponding grads-edge, then uses a designed length-variable l D template to scan the light-stripes＇ grads-edges. The template is able to find the disturbances with different width utilizing the distributing character of the edge disturbances. The found disturbances are eliminated finally. The algorithm not only can smoothly segment the light-stripes images, but also eliminate most disturbances on the light-stripes＇ edges without damaging the light-stripes images＇ 3D information. A practical example of using the proposed algorithm is given in the end. It is proved that the efficiency of the algorithm has been improved obviously by comparison.

Three-Dimensional Measurement Using Structured Light Based on Deep Learning

Three-dimensional(3D)reconstruction using structured light projection has the characteristics of non-contact,high precision,easy operation,and strong real-time performance.However,for actual measurement,projection modulated images are disturbed by electronic noise or other interference,which reduces the precision of the measurement system.To solve this problem,a 3D measurement algorithm of structured light based on deep learning is proposed.The end-to-end multi-convolution neural network model is designed to separately extract the coarse-and fine-layer features of a 3D image.The point-cloud model is obtained by nonlinear regression.The weighting coefficient loss function is introduced to the multi-convolution neural network,and the point-cloud data are continuously optimized to obtain the 3D reconstruction model.To verify the effectiveness of the method,image datasets of different 3D gypsum models were collected,trained,and tested using the above method.Experimental results show that the algorithm effectively eliminates external light environmental interference,avoids the influence of object shape,and achieves higher stability and precision.The proposed method is proved to be effective for regular objects.

MULTIPARAMETER MEASUREMENT FOR RACEWAY GROOVE OF BEARING BASED ON 3D RECONSTRUCTION WITH DIGITAL STRUCTURED LIGHT

A fast 3D reconstruction method based on structured light to measure various parameters of the raceway groove is presented. Digital parallel grating stripes distributed with sine density are projected onto the raceway groove by a DLP projector, and distorting of stripes is happened on the raceway. Simultaneously, aided by three-step phase-shifting approach, three images covered by different stripes are obtained by a high-resolution CCD camera at the same location, thus a more accuracy local topography can be obtained. And then the bearing is rotated on a high precision computer controlled rotational stage. Three images are also obtained as the former step at next planned location triggered by the motor. After one cycle, all images information is combined through the mosaics. As a result, the 3D information of raceway groove can be gained. Not only geometric properties but also surface flaws can be extracted by software. A preliminary hardware system has been built, with which some geometric parameters have been extracted from reconstructed local topography.

Application of color structured light pattern to measurement of large out-of-plane deformation

Measurement of out-of-plane deformation is significant to understanding of the deflection mechanisms of the plate and tube structures.In this study,a new surface contouring technique with color structured light is applied to measure the out-of-plane deformation of structures with one-shot projection.Through color fringe recognizing,decoding and triangulation processing for the captured images corresponding to each deformation state,the feasibility of the method is testified by the measurement of elastic deflections of a flexible square plate,showing good agreement with those from the calibrated displacement driver.The plastic deformation of two alloy aluminum rectangular tubes is measured to show the technique application to surface topographic evaluation of the buckling structures with large displacements.

Filtering algorithm of line structured light for long-distance obstacle detection

Since unmanned ground vehicles often encounter concave and convex obstacles in wild ground, a filtering algorithm using line structured light to detect these long distance obstacles is proposed. For the line structured light image, a ranked-order based adaptively extremum median （RAEM） filter algorithm on salt and pepper noise is presented. In the algorithm, firstly effective points and noise points in a filtering window are differentiated; then the gray values of noise points are replaced by the medium of gray values of the effective pixels, with the efficient points＇ gray values unchanged; in the end this algorithm is proved to be efficient by experiments. Experimental resuits demonstrate that the image blur, resulting into proposed algorithm can remove noise points effectively and minimize the protecting the edge information as much as possible.

Robust structured light in atmospheric turbulence

Structured light is routinely used in free-space optical communication channels,both classical and quantum,where information is encoded in the spatial structure of the mode for increased bandwidth.Both real-world and experimentally simulated turbulence conditions have revealed that free-space structured light modes are perturbed in some manner by turbulence,resulting in both amplitude and phase distortions,and consequently,much attention has focused on whether one mode type is more robust than another,but with seemingly inconclusive and contradictory results.We present complex forms of structured light that are invariant under propagation through the atmosphere:the true eigenmodes of atmospheric turbulence.We provide a theoretical procedure for obtaining these eigenmodes and confirm their invariance both numerically and experimentally.Although we have demonstrated the approach on atmospheric turbulence,its generality allows it to be extended to other channels too,such as aberrated paths,underwater,and in optical fiber.

High Curvature Stripe Profile Extraction Algorithm of Line Structured Light Measuring System

In the line structured light measuring system, the accuracy of the process of laser stripe directly affects the measurement results. Therefore, the extraction algorithm for the laser stripe, especially the surface with high reflection and high curvature, is very important. The imaging principle of line structured light, the light intensity distribution law of laser stripe and the extraction algorithm have been studied, and a stripe profile extraction method based on real light intensity distribution has been proposed. In this algorithm, fast region of interest extraction, stripe width estimation, and adaptive filtering on the striped image are performed. Then the energy center of the stripe at the sub-pixel level is extracted. Finally, the low-quality center points are eliminated, and the context information is used to recover the missing central points. Simulated images generated based on the imaging principle of line structured light and real experimental images were used to evaluate the accuracy and repeatability of the proposed method. The results show that the method behaves excellently at the edges of high-curvature stripes;the maximum error is only 1.6 pixels, which is 1/10 of the classic Steger algorithm;the experiment repeatability is only 8.8 μm, which is 2.7 times that of the Steger method. Therefore, the proposed method improves the accuracy of object contour extraction, and it is especially suitable for contour detection of objects with high curvature.

Dynamically measuring the holo-information of light fields in three-dimensional space using a periodic polarization-structured light

Spatially structured light field has attracted great attention due to its novel properties and application potential in numerous fields.Among them,the most striking one is the polarization-structured light,known as the vector beam.Here,using a periodic polarization-structured light,we propose a method to dynamically measure the holo-information of light fields,including the amplitude,phase,and polarization distributions,in three-dimensional(3D)space.The measurement system is composed of a Mach-Zender interferometer involving a liquid crystal polarized grating in the reference arm,which is simple,stable,and easy to operate.Featuring the single-shot measurement,this method supports observing the dynamic variation of object light fields.The accuracy,3D polarimetry,and dynamic observation of this method are validated by measuring a calibrated quarter-wave plate,a vector vortex beam,a Poincarébeam,and a stressed polymethyl methacrylate sample.

Three-line structured light vision system for non-cooperative satellites in proximity operations

Adapter ring is a commonly used component in non-cooperative satellites,which has high strength and is suitable to be recognized and grasped by the space manipulator.During proximity operations,this circle feature may be occluded by the robot arm or limited field of view.Moreover,the captured images may be underexposed when there is not enough illumination.To address these problems,this paper presents a structured light vision system with three line lasers and a monocular camera.The lasers project lines onto the surface of the satellite,and six break points are formed along both sides of the adapter ring.A closed-form solution for real-time pose estimation is given using these break points.Then,a virtual structured light platform is constructed to simulate synthetic images of the target satellite.Compared with the predefined camera parameters and relative positions,the proposed method is demonstrated to be more effective,especially at a close distance.Besides,a physical space verification system is set up to prove the effectiveness and robustness of our method under different light conditions.Experimental results indicate that it is a practical and effective method for the pose measurement of on-orbit tasks.

Metasurfaces enabling structured light manipulation:advances and perspectives [Invited]

Metasurfaces and structured light have rapidly advanced over the past few years, from being paradigms to forming functional devices and tailoring special light beams for wide emerging applications. Here, we focus on harnessing metasurfaces for structured light manipulation. We review recent advances in shaping structured light by metasurfaces on different platforms（metal, silica, silicon, and fiber）. Structured light manipulation based on plasmonic metasurfaces, reflection-enhanced plasmonic metasurfaces, metasurfaces on fiber facets, dielectric metasurfaces, and sub-wavelength structures on silicon are presented, showing impressive performance.Future trends, challenges, perspectives, and opportunities are also discussed.

Surface Reconstruction by Structured Light Projection

A rapid and practical method is proposed to reconstruct surface based on the linked structured light stripes which are produced by structured light projection.The subpixel points on a stripe are linked firstly one by one to form a stripe ensemble which is then transformed to a point ensemble in 3D space.The initial mesh with local optimization is generated by triangulating each two adjacent point ensembles.In order to obtain a better mesh,our improved edge flipping algorithm is employed to optimize the initial mesh globally.Because of employing the information of the linked structured stripes,our reconstruction algorithm is performed fastly.Moreover,the subpixel points on each stripe are already linked on the captured images such that they do not require the high sampling density.The experiments show that the proposed method constructs a surface rapidly and effectively.

Viewpoint Planning for Freeform Surface Inspection Using Plane Structured Light Scanners

This paper proposes an automatic model-based viewpoint planning method, which can achieve high precision and high efficiency for freeform surfaces inspection using plane structured light scanners. The surface model is utilized in stereolithography format, which is widely used as an industrial standard. The proposed method consists of 4 steps： topology reconstruction, mesh refinement, scan direction determination and viewpoint generation. In the first step, the topology structure of the surface model is reconstructed according to a designed data structure, based on which a neighborhood search algorithm is developed. In the second step, big facets in the surface model are segmented into several small ones, which are suitable for viewpoint planning. In the third step, an initial scan region of a viewpoint is grouped by the neighborhood search algorithm combining with total area and normal vector restrictions. Accordingly, the scan direction is determined by the normal vectors of facets in the initial scan region. In the fourth step, the position, the orientation, and the final scan region of the viewpoint are determined by 4 scan constraints, i.e., field of view, working distance range, view angle and overlap. Experimental results verify the effectiveness and advantages of the proposed method.

A review of liquid crystal spatial light modulators:devices and applications

Spatial light modulators,as dynamic flat-panel optical devices,have witnessed rapid development over the past two decades,concomitant with the advancements in micro-and opto-electronic integration technology.In particular,liquid-crystal spatial light modulator(LC-SLM)technologies have been regarded as versatile tools for generating arbitrary optical fields and tailoring all degrees of freedom beyond just phase and amplitude.These devices have gained significant interest in the nascent field of structured light in space and time,facilitated by their ease of use and real-time light manipulation,fueling both fundamental research and practical applications.Here we provide an overview of the key working principles of LC-SLMs and review the significant progress made to date in their deployment for various applications,covering topics as diverse as beam shaping and steering,holography,optical trapping and tweezers,measurement,wavefront coding,optical vortex,and quantum optics.Finally,we conclude with an outlook on the potential opportunities and technical challenges in this rapidly developing field.

Generation of structure light in probe absorption spectrum via microwave-driven Y-type atomic system

Behavior of structure light is investigated by monitoring probe absorption using a microwave-driven Y-type atomic media configuration.The system under consideration is driven by one of the control vortex beams as well as an extra non-vortex control beam to ensure electromagnetically induced transparency.The significant aspect in the generation of structured light is the azimuthal phase-dependent modification for probe absorption.Further intensity distribution for absorption spectra is examined for simultaneously evaluating both the control vortex beams.We also go through the radial distribution of intensity for various orbital angular momentum values.Different modes of structural beams may be distinguished using the suggested approach.Our research gives us a way for rapidly transferring vortex wavefronts from control field to probe absorption profile.This could be useful in quantum information processing.