Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Transient Stability Preventive Control of Wind Farm Connected Power System Considering the Uncertainty

To address uncertainty as well as transient stability constraints simultaneously in the preventive control of windfarm systems, a novel three-stage optimization strategy is established in this paper. In the first stage, the probabilisticmulti-objective particle swarm optimization based on the point estimate method is employed to cope with thestochastic factors. The transient security region of the system is accurately ensured by the interior point methodin the second stage. Finally, the verification of the final optimal objectives and satisfied constraints are enforcedin the last stage. Furthermore, the proposed strategy is a general framework that can combine other optimizationalgorithms. The proposed methodology is tested on the modified WSCC 9-bus system and the New England 39-bussystem. The results verify the feasibility of the method.

Development, Delivery and Evaluation of a Safety Training Program on Demolition for Blight Reduction

The main objective of the work presented in this paper was to develop a customized safety training program that can be incorporated into the demolition projects undertaken as part of blight reduction efforts in urban centers. A subsidiary objective was to devise and implement a safety program evaluation methodology, and gain insights on the relationships between knowledge acquisition through training and trainee demographics. Salient aspects of blight elimination efforts, as well as the main facets of building demolition practices and requirements, were reviewed. Information on various related safety and health hazards was studied in depth with a focus on demolition operations dealing with blighted properties. A unique safety hazard awareness training program was created for demolition workers, contractors and inspectors based on this research. In addition to devising a curriculum of relevant training topics along with traditional and online delivery systems to be employed, effectiveness evaluation instruments were formulated. Based on the limited data collected from the trainees it was concluded that the program was well-received by them and provided effective learning. It was also found that no statistically significant associations existed between the knowledge gain of the trainees, and either their experience level or union status, after taking this training.

Event-based performance guaranteed tracking control for constrained nonlinear system via adaptive dynamic programming method

An optimal tracking control problem for a class of nonlinear systems with guaranteed performance and asymmetric input constraints is discussed in this paper.The control policy is implemented by adaptive dynamic programming(ADP)algorithm under two event-based triggering mechanisms.It is often challenging to design an optimal control law due to the system deviation caused by asymmetric input constraints.First,a prescribed performance control technique is employed to guarantee the tracking errors within predetermined boundaries.Subsequently,considering the asymmetric input constraints,a discounted non-quadratic cost function is introduced.Moreover,in order to reduce controller updates,an event-triggered control law is developed for ADP algorithm.After that,to further simplify the complexity of controller design,this work is extended to a self-triggered case for relaxing the need for continuous signal monitoring by hardware devices.By employing the Lyapunov method,the uniform ultimate boundedness of all signals is proved to be guaranteed.Finally,a simulation example on a mass–spring–damper system subject to asymmetric input constraints is provided to validate the effectiveness of the proposed control scheme.

Policy Iteration for Optimal Control of Discrete-Time Time-Varying Nonlinear Systems

Aimed at infinite horizon optimal control problems of discrete time-varying nonlinear systems,in this paper,a new iterative adaptive dynamic programming algorithm,which is the discrete-time time-varying policy iteration(DTTV)algorithm,is developed.The iterative control law is designed to update the iterative value function which approximates the index function of optimal performance.The admissibility of the iterative control law is analyzed.The results show that the iterative value function is non-increasingly convergent to the Bellman-equation optimal solution.To implement the algorithm,neural networks are employed and a new implementation structure is established,which avoids solving the generalized Bellman equation in each iteration.Finally,the optimal control laws for torsional pendulum and inverted pendulum systems are obtained by using the DTTV policy iteration algorithm,where the mass and pendulum bar length are permitted to be time-varying parameters.The effectiveness of the developed method is illustrated by numerical results and comparisons.

Adaptive Multi-Step Evaluation Design With Stability Guarantee for Discrete-Time Optimal Learning Control

This paper is concerned with a novel integrated multi-step heuristic dynamic programming(MsHDP)algorithm for solving optimal control problems.It is shown that,initialized by the zero cost function,MsHDP can converge to the optimal solution of the Hamilton-Jacobi-Bellman(HJB)equation.Then,the stability of the system is analyzed using control policies generated by MsHDP.Also,a general stability criterion is designed to determine the admissibility of the current control policy.That is,the criterion is applicable not only to traditional value iteration and policy iteration but also to MsHDP.Further,based on the convergence and the stability criterion,the integrated MsHDP algorithm using immature control policies is developed to accelerate learning efficiency greatly.Besides,actor-critic is utilized to implement the integrated MsHDP scheme,where neural networks are used to evaluate and improve the iterative policy as the parameter architecture.Finally,two simulation examples are given to demonstrate that the learning effectiveness of the integrated MsHDP scheme surpasses those of other fixed or integrated methods.

Parallel Control for Optimal Tracking via Adaptive Dynamic Programming

This paper studies the problem of optimal parallel tracking control for continuous-time general nonlinear systems.Unlike existing optimal state feedback control,the control input of the optimal parallel control is introduced into the feedback system.However,due to the introduction of control input into the feedback system,the optimal state feedback control methods can not be applied directly.To address this problem,an augmented system and an augmented performance index function are proposed firstly.Thus,the general nonlinear system is transformed into an affine nonlinear system.The difference between the optimal parallel control and the optimal state feedback control is analyzed theoretically.It is proven that the optimal parallel control with the augmented performance index function can be seen as the suboptimal state feedback control with the traditional performance index function.Moreover,an adaptive dynamic programming(ADP)technique is utilized to implement the optimal parallel tracking control using a critic neural network(NN)to approximate the value function online.The stability analysis of the closed-loop system is performed using the Lyapunov theory,and the tracking error and NN weights errors are uniformly ultimately bounded(UUB).Also,the optimal parallel controller guarantees the continuity of the control input under the circumstance that there are finite jump discontinuities in the reference signals.Finally,the effectiveness of the developed optimal parallel control method is verified in two cases.

Optimal Control for a Class of Complex Singular System Based on Adaptive Dynamic Programming

This paper presents a new design approach to achieve decentralized optimal control of high-dimension complex singular systems with dynamic uncertainties. Based on robust adaptive dynamic programming(robust ADP) method, controllers for solving the singular systems optimal control problem are designed. The proposed algorithm can work well when the system model is not exactly known but the input and output data can be measured. The policy iteration of each controller only uses their own states and input information for learning,and do not need to know the whole system dynamics. Simulation results on the New England 10-machine 39-bus test system show the effectiveness of the designed controller.

Fuzzy neural network control of underwater vehicles based on desired state programming

Due to the nonlinearity and uncertainty, the precise control of underwater vehicles in some intelligent operations hasn’t been solved very well yet. A novel method of control based on desired state programming was presented, which used the technique of fuzzy neural network. The structure of fuzzy neural network was constructed according to the moving characters and the back propagation algorithm was deduced. Simulation experiments were conducted on general detection remotely operated vehicle. The results show that there is a great improvement in response and precision over traditional control, and good robustness to the model’s uncertainty and external disturbance, which has theoretical and practical value.

Program structures and synchronization schemes in numerical coordinate controllers of three-wire welding

In the three-wire welding system, a welding process consists of the operations of four devices, namely three welding machines and one bogie. The operations need to be synchronized by a numerical coordinate controller （ NCC ）. In this paper, we will discuss a tnsk-job-procedure cubic program structure. Under this structure, the devices are synchronized and isolated at the same time. This cubic program structure can also be used as a reference for other multi-device or multi-unit manufacturing processes.

CONTROLLING ROBOT MANIPULATORS BY DYNAMIC PROGRAMMING

A certain number of considerations should be taken into account in the dynamic control of robot manipulators as highly complex non-linear systems.In this article,we provide a detailed presentation of the mechanical and electrical impli- cations of robots equipped with DC motor actuators.This model takes into account all non-linear aspects of the system.Then,we develop computational algorithms for optimal control based on dynamic programming.The robot's trajectory must be predefined,but performance criteria and constraints applying to the system are not limited and we may adapt them freely to the robot and the task being studied.As an example,a manipulator arm with 3 degrees of freedom is analyzed.

Stability of Nonlinear Systems Using Optimal Fuzzy Controllers and Its Simulation by Java Programming

In this paper, at first, the single input rule modules(SIRMs) dynamically connected fuzzy inference model is used to stabilize a double inverted pendulum system. Then, a multiobjective particle swarm optimization(MOPSO) is implemented to optimize the fuzzy controller parameters in order to decrease the distance error of the cart and summation of the angle errors of the pendulums, simultaneously. The feasibility and efficiency of the proposed Pareto front is assessed in comparison with results reported in literature and obtained from other algorithms.Finally, the Java programming with applets is utilized to simulate the stability of the nonlinear system and explain the internetbased control.

Assessment of the crucial factors influencing the responses of ammonia and nitrous oxide emissions to controlled release nitrogen fertilizer: A meta-analysis

Reducing ammonia(NH3) and nitrous oxide(N2O) emissions have great effects on mitigating nitrogen(N) nutrient loss and greenhouse gas emissions. Controlled release urea(CRU) can control the N release rate, which reduces reactive N loss and increases nitrogen use efficiency relative to conventional urea(CU). However, the crucial factors influencing the responses of NH3and N2O emissions to CRU relative to CU are still unclear. In this study, we evaluated the responses of NH3and N2O emissions to CRU based on collected field data with a meta-analysis. CRU reduced the NH3and N2O emissions by 32.7 and 25.0% compared with CU, respectively. According to subgroup analysis, CRU presented better mitigation of NH3and N2O emissions in soils with pH 6.5–7.5(–47.9 and –23.7%) relative to either pH<6.5(–28.5and –21.4%) or pH>7.5(–29.3 and –17.3%), and in the rice season(–34.8 and –29.1%) relative to the wheat season(–19.8 and –22.8%). The responses of NH3and N2O emissions to CRU increased from rainfed(–30.5 and –17.0%) to irrigated(–32.5 and –22.9%), and then to paddy(–34.8 and –29.1%) systems. In addition, the response of N2O emission mitigation increased with increases in soil total nitrogen(TN);however, soil TN did not significantly affect the response of NH3volatilization. The reduction in NH3emission was greater in sandy-textured soil(–57.7%) relative to loam-textured(–32.9%) and clay-textured(–32.3%) soils, whereas soil texture did not affect N2O emission. Overall, CRU was a good option for reducing the NH3and N2O emissions relative to CU in agricultural production. This analysis improves our understanding of the crucial environmental and management factors influencing the mitigation of NH3and N2O emissions under CRU application, and these site-specific factors should be considered when applying CRU to reduce reactive N loss and increase NUE.

Linear Programming for Optimum PID Controller Tuning

This work presents a new methodology based on Linear Programming (LP) to tune Proportional-Integral-Derivative (PID) control parameters. From a specification of a desired output time domain of the plant, a linear optimization system is proposed to adjust the PID controller leading the output signal to stable operation condition with minimum oscillations. The constraint set used in the optimization process is defined by using numerical integration approach. The generated optimization problem is convex and easily solved using an interior point algorithm. Results obtained using familiar plants from literature have shown that the proposed linear programming problem is very effective for tuning PID controllers.

Using Automation Controller System and Simulation Program for Testing Closed Circuits of Mini-Sprinkler Irrigation System

The field experiments were conducted at the experimental farm of Faculty of agricultural, southern Illinois University SIUC, USA. The project makes the irrigation automated. With the use of low cost sensors and the simple circuitry makes currently project a low cost product, which can be bought even by a poor farmer. This research work is best suited for places where water is scares and has to be used in limited quantity and this proposal is a model to modernize the agriculture industries at a mass scale with optimum expenditure. In the field of agricultural engineering, use of sensor method of irrigation operation is important and it is well known that closed circuits of Mini-sprinkler irrigation system are very economical and efficient. Closed circuits are considered one of the modifications of Mini-sprinkler irrigation system, and added advantages to Mini-sprinkler irrigation system because it can relieve low operating pressures problem at the end of the lateral lines. In the conventional closed circuits of Mini-sprinkler irrigation system, the farmer has to keep watch on irrigation timetable, which is different for different crops. Using this system, one can save manpower, water to improve production and ultimately profit. The data could be summarized in following: Irrigation methods under study when using lateral length 60 mcould be ranked in the following ascending order according the values of the predicted and measured head losses CM1M-SIS CM2M-SIS.The correlation (Corr.) coefficients were used to compare the predicted and measured head losses along the lateral lines of all the closed circuits designs. Generally, the values of correlation analysis were (>0.90) were obtained with 0% field slope60 mlength (experimental conditions) for all closed circuits.The interaction between irrigation methods: at the start there are significant differences between CM2M-SIS and CM1M-SIS.

Pollution Control Programs of Large-Scale Livestock and Poultry Farms in the Dianchi Lake Basin

With the enlarging scale and intensifying production of livestock and poultry breeding, the environment pollution becomes increasingly prominent in the Dianchi Lake Basin since 1990s. According to the survey of ＂The First National Census of Pollution Sources＂, occurrence and discharge of pollutants in large-scale livestock and poultry farms in this region were first understood. The pollution characteristics of large-scale live- stock and poultry breeding were also analyzed deeply. On this basis, the significance of pollution control programs for environment protection was investigated from aspects of pollution control policy, technology management and publicity.

Tri-level programming model for combined urban traffic signal control and traffic flow guidance

In order to balance the temporal-spatial distribution of urban traffic flow, a model is established for combined urban traffic signal control and traffic flow guidance. With consideration of the wide use of fixed signal control at intersections, traffic assignment under traffic flow guidance, and dynamic characteristics of urban traffic management, a tri-level programming model is presented. To reflect the impact of intersection delay on traffic assignment, the lower level model is set as a modified user equilibrium model. The middle level model, which contains several definitional constraints for different phase modes, is built for the traffic signal control optimization. To solve the problem of tide lane management, the upper level model is built up based on nonlinear 0-1 integer programming. A heuristic iterative optimization algorithm(HIOA) is set up to solve the tri-level programming model. The lower level model is solved by method of successive averages(MSA), the middle level model is solved by non-dominated sorting genetic algorithm II(NSGA II), and the upper level model is solved by genetic algorithm(GA). A case study is raised to show the efficiency and applicability of the proposed modelling and computing method.

Spatial Control of Lithium Deposition by Controlling the Lithiophilicity with Copper(Ⅰ)Oxide Boundaries

Spatial control of lithium deposition is the most important issue in lithium-metal batteries because of the considerable control of lithium dendrite suppression via the uniform distribution of Li^(+)flux.Although seed materials are crucial for the behavior of lithium deposition,in-depth studies on their physical and chemical control have not been conducted.Here,we describe a new design of seed structure comprising a wrinkled Cu/graphene substrate surrounded by copper(Ⅰ)oxide(Cu_(2)O)on a graphene grain boundary over a large area,which is fabricated by the oxidation of the Cu surface via graphene boundary defects by using chemical vapor deposition(CVD).Scanning and transmission electron microscopy results reveal that Cu_(2)O on the graphene boundary can render a preferential reaction with lithium during the first deposition and assist in the uniform deposition of lithium by preventing the agglomeration of lithium clusters during the second deposition.This two-step process is attributed to the degree of selectivity due to the difference in lithium affinity,which allows long-term electrochemical stability and a high rate capability via boundary effects.This study highlights the significance of the boundary effect,which can open new avenues for the formation of a large family of seed structures in lithium-metal batteries.

Approximate Dynamic Programming for Self-Learning Control

This paper introduces a self-learning control approach based on approximate dynamic programming. Dynamic programming was introduced by Bellman in the 1950's for solving optimal control problems of nonlinear dynamical systems. Due to its high computational complexity, the applications of dynamic programming have been limited to simple and small problems. The key step in finding approximate solutions to dynamic programming is to estimate the performance index in dynamic programming. The optimal control signal can then be determined by minimizing (or maximizing) the performance index. Artificial neural networks are very efficient tools in representing the performance index in dynamic programming. This paper assumes the use of neural networks for estimating the performance index in dynamic programming and for generating optimal control signals, thus to achieve optimal control through self-learning.

基于国产SOPC的多通道控制模块设计

针对某火炮对击发控制的高可靠性和准确性需求,设计了一种基于国产SOPC的多路信号采集和控制模块,应用于其击发控制。设计了以SOPC为核心控制器的信号采集和控制电路,其中SOPC微内核与内部FPGA之间为双向实时通信机制,硬件采用Verilog HDL对SOPC进行了逻辑设计和外设电路构架,实现了多信号采集和控制,利用内部微处理器完成与外部设备的通信和数据交互,依托仿真测试平台完成了SOPC的通信时序和功能验证,并利用示波器和上位机进行了测试。测试结果表明,从输入信号满足条件到输出相关信号的时延低至250μs,多次测试的时延误差不超过±10μs,而采用PLC控制器进行击发控制则需要3~8 ms,较好地满足了击发控制的准确性和可靠性要求。