Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

An intelligent control method based on artificial neural network for numerical flight simulation of the basic finner projectile with pitching maneuver

In this paper,an intelligent control method applying on numerical virtual flight is proposed.The proposed algorithm is verified and evaluated by combining with the case of the basic finner projectile model and shows a good application prospect.Firstly,a numerical virtual flight simulation model based on overlapping dynamic mesh technology is constructed.In order to verify the accuracy of the dynamic grid technology and the calculation of unsteady flow,a numerical simulation of the basic finner projectile without control is carried out.The simulation results are in good agreement with the experiment data which shows that the algorithm used in this paper can also be used in the design and evaluation of the intelligent controller in the numerical virtual flight simulation.Secondly,combined with the real-time control requirements of aerodynamic,attitude and displacement parameters of the projectile during the flight process,the numerical simulations of the basic finner projectile’s pitch channel are carried out under the traditional PID(Proportional-Integral-Derivative)control strategy and the intelligent PID control strategy respectively.The intelligent PID controller based on BP(Back Propagation)neural network can realize online learning and self-optimization of control parameters according to the acquired real-time flight parameters.Compared with the traditional PID controller,the concerned control variable overshoot,rise time,transition time and steady state error and other performance indicators have been greatly improved,and the higher the learning efficiency or the inertia coefficient,the faster the system,the larger the overshoot,and the smaller the stability error.The intelligent control method applying on numerical virtual flight is capable of solving the complicated unsteady motion and flow with the intelligent PID control strategy and has a strong promotion to engineering application.

Analysis of debris flow control effect and hazard assessment in Xinqiao Gully,Wenchuan M_(s)8.0 earthquake area based on numerical simulation

Xinqiao Gully is located in the area of the 2008 Wenchuan M_(s)8.0 earthquake in Sichuan province,China.Based on the investigation of the 2023"6-26"Xinqiao Gully debris flow event,this study assessed the effectiveness of the debris flow control project and evaluated the debris flow hazards.Through field investigation and numerical simulation methods,the indicators of flow intensity reduction rate and storage capacity fullness were proposed to quantify the effectiveness of the engineering measures in the debris flow event.The simulation results show that the debris flow control project reduced the flow intensity by41.05%to 64.61%.The storage capacity of the dam decreases gradually from upstream to the mouth of the gully,thus effectively intercepting and controlling the debris flow.By evaluating the debris flow of different recurrence intervals,further measures are recommended for managing debris flow events.

Event-based performance guaranteed tracking control for constrained nonlinear system via adaptive dynamic programming method

An optimal tracking control problem for a class of nonlinear systems with guaranteed performance and asymmetric input constraints is discussed in this paper.The control policy is implemented by adaptive dynamic programming(ADP)algorithm under two event-based triggering mechanisms.It is often challenging to design an optimal control law due to the system deviation caused by asymmetric input constraints.First,a prescribed performance control technique is employed to guarantee the tracking errors within predetermined boundaries.Subsequently,considering the asymmetric input constraints,a discounted non-quadratic cost function is introduced.Moreover,in order to reduce controller updates,an event-triggered control law is developed for ADP algorithm.After that,to further simplify the complexity of controller design,this work is extended to a self-triggered case for relaxing the need for continuous signal monitoring by hardware devices.By employing the Lyapunov method,the uniform ultimate boundedness of all signals is proved to be guaranteed.Finally,a simulation example on a mass–spring–damper system subject to asymmetric input constraints is provided to validate the effectiveness of the proposed control scheme.

Application of numerical modeling and genetic programming to estimate rock mass modulus of deformation

Estimation of the rock mass modulus of deformation(Em)is one of the most important design parameters in designing many structures in and on rock.This parameter can be obtained by in situ tests,empirical relations between deformation modulus and rock mass classifcation,and estimating from laboratory tests results.In this paper,a back analysis calculation is performed to present an equation for estimation of the rock mass modulus of deformation using genetic programming(GP)and numerical modeling.A database of 40,960 datasets,including vertical stress(rz),horizontal to vertical stresses ratio(k),Poisson’s ratio(m),radius of circular tunnel(r)and wall displacement of circular tunnel on the horizontal diameter(d)for input parameters and modulus of deformation for output,was established.The selected parameters are easy to determine and rock mass modulus of deformation can be obtained from instrumentation data of any size circular galleries.The resulting RMSE of 0.86 and correlation coeffcient of97%of the proposed equation demonstrated the capability of the computer program(CP)generated by GP.

Java Language for Numerical Control Simulation System Research

Java language is widely used in a variety of development platforms to develop all kinds of application software for its simple and efficient.The programming language owning platform independent is adopted to solve the image flicker,sound loading and other issues by the threading technology,multimedia technology,graphics and point by point comparison techniques.Dynamic real⁃time process simulation is implemented and a two⁃dimensional network of CNC machining simulation system is developed while Java applet application is as a carrier.The simulation results show that the system has a friendly interface and fast calculation speed,and platform portability has a certain practicality and application value.

Program structures and synchronization schemes in numerical coordinate controllers of three-wire welding

In the three-wire welding system, a welding process consists of the operations of four devices, namely three welding machines and one bogie. The operations need to be synchronized by a numerical coordinate controller （ NCC ）. In this paper, we will discuss a tnsk-job-procedure cubic program structure. Under this structure, the devices are synchronized and isolated at the same time. This cubic program structure can also be used as a reference for other multi-device or multi-unit manufacturing processes.

Basic Numerical Computational Using Scilab Programming

Mathematics is very important for the engineering and scientist but to make understand the mathematics is very difficult if without proper tools and suitable measurement. A numerical method is one of the algorithms which involved with computer programming. In this paper, Scilab is used to carter the problems related the mathematical models such as Matrices, operation with ODE＇s and solving the Integration.

A NEW DETERMINISTIC FORMULATION FOR DYNAMIC STOCHASTIC PROGRAMMING PROBLEMS AND ITS NUMERICAL COMPARISON WITH OTHERS

A new deterministic formulation,called the conditional expectation formulation,is proposed for dynamic stochastic programming problems in order to overcome some disadvantages of existing deterministic formulations.We then check the impact of the new deterministic formulation and other two deterministic formulations on the corresponding problem size,nonzero elements and solution time by solving some typical dynamic stochastic programming problems with different interior point algorithms.Numerical results show the advantage and application of the new deterministic formulation.

Effect of geometrical parameters on forming quality of high-strength TA18 titanium alloy tube in numerical control bending

The forming quality of high-strength TA18 titanium alloy tube during numerical control bending in changing bending angle β, relative bending radius R/D and tube sizes such as diameter D and wall thickness t was clarified by finite element simulation. The results show that the distribution of wall thickness change ratio Δt and cross section deformation ratio ΔD are very similar under different β; the Δt and ΔD decrease with the increase of R/D, and to obtain the qualified bent tube, the R/D must be greater than 2.0; the wall thinning ratio Δto slightly increases with larger D and t, while the wall thickening ratio Δti and ΔD increase with the larger D and smaller t; the Δto and ΔD firstly decrease and then increase, while the Δti increases, for the same D/t with the increase of D and t.

METHOD FOR SUPPRESSING CUTTING CHATTER IN NUMERICAL CONTROL MACHINE TOOLS

A new method for suppressing cutting chatter is studied by adjusting servo parameters of the numerical control （NC） machine tool and controlling the limited cutting width. A model of the cutting system of the NC machine tool is established. It includes the mechanical system, the servo system and the cutting chatter system. Interactions between every two systems are shown in the model. The cutting system stability is simulated and relation curves between the limited cutting width and servo system parameters are described in the experiment. Simulation and experimental results show that there is a mapping relation between the limited cutting width and servo parameters of the NC machine tool, and the method is applicable and credible to suppress chatter.

Shaking table test and numerical analysis of offshore wind turbine tower systems controlled by TLCD

A wind turbine system equipped with a tuned liquid column damper （TLCD） is comprehensively studied via shaking table tests using a 1/13-scaled model. The effects of wind and wave actions are considered by inputting response- equivalent accelerations on the shaking table. The test results show that the control effect of the TLCD system is significant in reducing the responses under both wind-wave equivalent loads and ground motions, but obviously varies for different inputs, Further, a blade-hub-tower integrated numerical model for the wind turbine system is established. The model is capable of considering the rotational effect of blades by combining Kane＇s equation with the finite element method. The responses of the wind tower equipped with TLCD devices are numerically obtained and compared to the test results, showing that under both controlled and uncontrolled conditions with and without blades＇ rotation, the corresponding responses exhibit good agreement. This demonstrates that the proposed numerical model performs well in capturing the wind-wave coupled response of the offshore wind turbine systems under control. Both numerical and experimental results show that the TLCD system can significantly reduce the structural response and thus improve the safety and serviceability of the offshore wind turbine tower systems. Additional issues that require further study are discussed.

Numerical modelling of structural controls on fluid flow and mineralization

This paper presents the results of a set of numerical models focussing on structural controls on hydrothermal mineralization. We first give an overview of natural phenomena of structurally-controlled ore formation and the background theory and mechanisms for such controls. We then provide the results of a group of simple 2D numerical models validated through comparison with Cu-vein structure observed near the Shilu Copper deposit （Yangchun, Guangdong Province, China） and finally a case study of 3D numerical modelling applied to the Hodgkinson Province in North Queensland （Australia）. Two modelling approaches, discrete deformation modelling and continuum coupled deformation and fluid flow modelling, are involved. The 2D model-derived patterns are remarkably consistent with the Cu-vein structure from the Shilu Copper deposit, and show that both modelling approaches can realistically simulate the mechanical behaviours of shear and dilatant fractures. The continuum coupled deformation and fluid flow model indicates that pattern of the Cu- veins near the Shilu deposit is the result of shear strain localization, development of dilation and fluid focussing into the dilatant fracture segments. The 3D case-study models （with deformation and fluid flow coupling） on the Hodgkinson Province generated a number of potential gold mineralization

Parallel Control for Optimal Tracking via Adaptive Dynamic Programming

This paper studies the problem of optimal parallel tracking control for continuous-time general nonlinear systems.Unlike existing optimal state feedback control,the control input of the optimal parallel control is introduced into the feedback system.However,due to the introduction of control input into the feedback system,the optimal state feedback control methods can not be applied directly.To address this problem,an augmented system and an augmented performance index function are proposed firstly.Thus,the general nonlinear system is transformed into an affine nonlinear system.The difference between the optimal parallel control and the optimal state feedback control is analyzed theoretically.It is proven that the optimal parallel control with the augmented performance index function can be seen as the suboptimal state feedback control with the traditional performance index function.Moreover,an adaptive dynamic programming(ADP)technique is utilized to implement the optimal parallel tracking control using a critic neural network(NN)to approximate the value function online.The stability analysis of the closed-loop system is performed using the Lyapunov theory,and the tracking error and NN weights errors are uniformly ultimately bounded(UUB).Also,the optimal parallel controller guarantees the continuity of the control input under the circumstance that there are finite jump discontinuities in the reference signals.Finally,the effectiveness of the developed optimal parallel control method is verified in two cases.

Underground ground control monitoring and interpretation,and numerical modeling, and shield capacity design

Mine or longwall panel layout is a 3D structure with highly non-uniform stress distribution. Recognition of such fact will facilitate underground problem identification/investigation and solving by numerical modeling through proper model construction. Due to its versatility, numerical modeling is the most popular method for ground control design and problem solving. However numerical modeling results require highly experienced professionals to interpret its validity/applicability to actual mining operations due to complicated mining and geological conditions. Underground ground control monitoring is routinely performed to predict roof behavior such as weighting and weighting interval without matching observation of face mining condition while the mining pressures are being monitored, resulting in unrealistic interpretation of the obtained data on mining pressure. The importance of ground control pressure monitoring and simultaneous observation of mining and geological conditions is illustrated by an example of shield leg pressure monitoring and interpretation in an U.S. longwall coal mine: it was found that the roof strata act like a plate, not an individual block of the size of a shield dimension, as commonly assumed by all researchers and shield capacity is not a fixed property for a longwall panel or a mine or a coal seam. A new mechanism on the interaction between shield's hydraulic leg pressure and roof strata for shield loading is proposed.

NUMERICAL SOLUTION OF VISCOUS FLOW PAST A SOLID SPHERE WITH THE CONTROL VOLUME FORMULATION

The control volume formulation with the QUICK finite difference scheme is used to solveincompressible liquid flow past a solid sphere in terms of stream function and vorticity.Several tech-nical points are addressed on improving the accuracy and efficiency of numerical simulation of similarproblems of fluid flow.In particular,the importance of suitable specification of the distortion func-tion to enforcing the far field boundarv conditions is emphasized.

NOVEL STRUCTURE FOR DISTRIBUTED NUMERICAL CONTROL

Because of an unexpected signal noise within the network or an unpredictedfault with personal computers (PCs), many problems emerge in the implementation of distributednumerical control (DNC) with PCs-based network. To solve the problems, an industrial solution ofinvolving the field-bus technology in DNC communicating area is provided. A kind of advancedField-bus, named controller area network (CAN), is originally developed to support cheap and rathersimple automotive applications. However, because of its good performance and low cost, it is alsobeing considered in automated-manufacturing and process control environments to interconnectintelligent devices, such as modem sensors and actuators. Recently it creates a new role for CANBusin DNC that brings new thinking to DNC. CAN is used as the network platform for connecting machinetools to share information with each other reliably. Additionally, thanks to also applying of'plug-in' technology and a special interface of hardware, this solution exhibits some highcompatibility with different pedigree numerical control (NC) systems, such as Fanuc, Siemens,Cincinnati and so on. In order to improve CANBus for DNC application, a communicating competitionmodel of the basic CAN protocol, called CC model, is then highlighted. This model is able to satisfythe requirements that different machine tools share the communicating bandwidth fairly when theyrun concurrently. Finally the novel view of the latest advancement in CANBus-based DNC incombination with the manufacturing paradigm is also presented.

Optimal Control for a Class of Complex Singular System Based on Adaptive Dynamic Programming

This paper presents a new design approach to achieve decentralized optimal control of high-dimension complex singular systems with dynamic uncertainties. Based on robust adaptive dynamic programming(robust ADP) method, controllers for solving the singular systems optimal control problem are designed. The proposed algorithm can work well when the system model is not exactly known but the input and output data can be measured. The policy iteration of each controller only uses their own states and input information for learning,and do not need to know the whole system dynamics. Simulation results on the New England 10-machine 39-bus test system show the effectiveness of the designed controller.

A Numerical-analytic Method for Quickly Predicting Springback of Numerical Control Bending of Thin-walled Tube

Springback is one of important factors influencing the forming quality of numerical control （NC） bending of thin-walled tube. In this paper, a numerical-analytic method for springback angle prediction of the process was put forward. The method is based on springback angle model derived using analytic method and simulation results from three-dimensional （3D） rigid-plastic finite element method （FEM）. The method is validated through comparison with experimental results. The features of the method are as follows： （1） The method is high in efficiency because it combines advantages of rigid-plastic FEM and analytic method. （2） The method is satisfactory in accuracy, since the field variables used in the model is resulting from 3D rigid-plastic FEM solution, and the effects both of axial force and strain neutral axis shift have been included. （3） Research on multi-factor effects can be carried out using the method due to its advantage inheriting from rigid-plastic FEM. The method described here is also of general significance to other bending processes.

A Windows bitmap-based numerically controlled sculpture method and its application

Bitmap （BMP） is a widely used format in image processing. With the combination of the digital image processing and computer numerically controlled （NC） techniques, we developed a bitmap-based NC single-step and multi-dot method. An example was provided to illustrate the application and principle of this bitmap-based method in our newly innovated sandblasting glass NC sculpture system.