Autonomous Vehicle Platoons In Urban Road Networks:A Joint Distributed Reinforcement Learning and Model Predictive Control Approach

In this paper, platoons of autonomous vehicles operating in urban road networks are considered. From a methodological point of view, the problem of interest consists of formally characterizing vehicle state trajectory tubes by means of routing decisions complying with traffic congestion criteria. To this end, a novel distributed control architecture is conceived by taking advantage of two methodologies: deep reinforcement learning and model predictive control. On one hand, the routing decisions are obtained by using a distributed reinforcement learning algorithm that exploits available traffic data at each road junction. On the other hand, a bank of model predictive controllers is in charge of computing the more adequate control action for each involved vehicle. Such tasks are here combined into a single framework:the deep reinforcement learning output(action) is translated into a set-point to be tracked by the model predictive controller;conversely, the current vehicle position, resulting from the application of the control move, is exploited by the deep reinforcement learning unit for improving its reliability. The main novelty of the proposed solution lies in its hybrid nature: on one hand it fully exploits deep reinforcement learning capabilities for decisionmaking purposes;on the other hand, time-varying hard constraints are always satisfied during the dynamical platoon evolution imposed by the computed routing decisions. To efficiently evaluate the performance of the proposed control architecture, a co-design procedure, involving the SUMO and MATLAB platforms, is implemented so that complex operating environments can be used, and the information coming from road maps(links,junctions, obstacles, semaphores, etc.) and vehicle state trajectories can be shared and exchanged. Finally by considering as operating scenario a real entire city block and a platoon of eleven vehicles described by double-integrator models, several simulations have been performed with the aim to put in light the main f eatures of the proposed approach. Moreover, it is important to underline that in different operating scenarios the proposed reinforcement learning scheme is capable of significantly reducing traffic congestion phenomena when compared with well-reputed competitors.

A Combined Reinforcement Learning and Model Predictive Control for Car-Following Maneuver of Autonomous Vehicles

Model predictive control is widely used in the design of autonomous driving algorithms.However,its parameters are sensitive to dynamically varying driving conditions,making it difficult to be implemented into practice.As a result,this study presents a self-learning algorithm based on reinforcement learning to tune a model predictive controller.Specifically,the proposed algorithm is used to extract features of dynamic traffic scenes and adjust the weight coefficients of the model predictive controller.In this method,a risk threshold model is proposed to classify the risk level of the scenes based on the scene features,and aid in the design of the reinforcement learning reward function and ultimately improve the adaptability of the model predictive controller to real-world scenarios.The proposed algorithm is compared to a pure model predictive controller in car-following case.According to the results,the proposed method enables autonomous vehicles to adjust the priority of performance indices reasonably in different scenarios according to risk variations,showing a good scenario adaptability with safety guaranteed.

Machine Learning Accelerated Real-Time Model Predictive Control for Power Systems

This paper presents a machine-learning-based speedup strategy for real-time implementation of model-predictive-control(MPC)in emergency voltage stabilization of power systems.Despite success in various applications,real-time implementation of MPC in power systems has not been successful due to the online control computation time required for large-sized complex systems,and in power systems,the computation time exceeds the available decision time used in practice by a large extent.This long-standing problem is addressed here by developing a novel MPC-based framework that i)computes an optimal strategy for nominal loads in an offline setting and adapts it for real-time scenarios by successive online control corrections at each control instant utilizing the latest measurements,and ii)employs a machine-learning based approach for the prediction of voltage trajectory and its sensitivity to control inputs,thereby accelerating the overall control computation by multiple times.Additionally,a realistic control coordination scheme among static var compensators(SVC),load-shedding(LS),and load tap-changers(LTC)is presented that incorporates the practical delayed actions of the LTCs.The performance of the proposed scheme is validated for IEEE 9-bus and 39-bus systems,with±20%variations in nominal loading conditions together with contingencies.We show that our proposed methodology speeds up the online computation by 20-fold,bringing it down to a practically feasible value(fraction of a second),making the MPC real-time and feasible for power system control for the first time.

Hybrid Dynamic Variables-Dependent Event-Triggered Fuzzy Model Predictive Control

This article focuses on dynamic event-triggered mechanism(DETM)-based model predictive control(MPC) for T-S fuzzy systems.A hybrid dynamic variables-dependent DETM is carefully devised,which includes a multiplicative dynamic variable and an additive dynamic variable.The addressed DETM-based fuzzy MPC issue is described as a “min-max” optimization problem(OP).To facilitate the co-design of the MPC controller and the weighting matrix of the DETM,an auxiliary OP is proposed based on a new Lyapunov function and a new robust positive invariant(RPI) set that contain the membership functions and the hybrid dynamic variables.A dynamic event-triggered fuzzy MPC algorithm is developed accordingly,whose recursive feasibility is analysed by employing the RPI set.With the designed controller,the involved fuzzy system is ensured to be asymptotically stable.Two examples show that the new DETM and DETM-based MPC algorithm have the advantages of reducing resource consumption while yielding the anticipated performance.

Iterative Learning Model Predictive Control for a Class of Continuous/Batch Processes

An iterative learning model predictive control （ILMPC） technique is applied to a class of continuous/batch processes. Such processes are characterized by the operations of batch processes generating periodic strong disturbances to the continuous processes and traditional regulatory controllers are unable to eliminate these periodic disturbances. ILMPC integrates the feature of iterative learning control （ILC） handling repetitive signal and the flexibility of model predictive control （MPC）. By on-line monitoring the operation status of batch processes, an event-driven iterative learning algorithm for batch repetitive disturbances is initiated and the soft constraints are adjusted timely as the feasible region is away from the desired operating zone. The results of an industrial application show that the proposed ILMPC method is effective for a class of continuous/batch processes.

A Novel Tuning Method for Predictive Control of VAV Air Conditioning System Based on Machine Learning and Improved PSO

The variable air volume(VAV)air conditioning system is with strong coupling and large time delay,for which model predictive control(MPC)is normally used to pursue performance improvement.Aiming at the difficulty of the parameter selection of VAV MPC controller which is difficult to make the system have a desired response,a novel tuning method based on machine learning and improved particle swarm optimization(PSO)is proposed.In this method,the relationship between MPC controller parameters and time domain performance indices is established via machine learning.Then the PSO is used to optimize MPC controller parameters to get better performance in terms of time domain indices.In addition,the PSO algorithm is further modified under the principle of population attenuation and event triggering to tune parameters of MPC and reduce the computation time of tuning method.Finally,the effectiveness of the proposed method is validated via a hardware-in-the-loop VAV system.

Parameter Self - Learning of Generalized Predictive Control Using BP Neural Network

This paper describes the self—adjustment of some tuning-knobs of the generalized predictive controller(GPC).A three feedforward neural network was utilized to on line learn two key tuning-knobs of GPC,and BP algorithm was used for the training of the linking-weights of the neural network.Hence it gets rid of the difficulty of choosing these tuning-knobs manually and provides easier condition for the wide applications of GPC on industrial plants.Simulation results illustrated the effectiveness of the method.

Fuzzy disturbance rejection predictive control of ultra-supercritical once-through boiler-turbine unit

In order to overcome the wide-range load tracking and unknown disturbance issues of an ultra-supercritical boiler- turbine unit, a fuzzy disturbance rejection predictive control approach is proposed using the techniques of fuzzy scheduling, model predictive control and extended state observer. Local state-space models are established on the basis of nonlinearity analysis and subspace identification. To eiJiance thedisturbance rejection capability of the controller, a extended state observer is employed to estimate unnown disturbances and model mismatches. The disturbance estimation ennaced local predictive controllers ae subsequently devised based on the local models, the performance of which is further strengthened by incorporating the fuzzy scheduling technique. The simulation results verify the merits of the proposed strategy in achieving satisfactory wide-range load tracking ad disturbance rejection performance.

Design and Analysis of Integrated Predictive Iterative Learning Control for Batch Process Based on Two-dimensional System Theory

Based on the two-dimensional （2D） system theory, an integrated predictive iterative learning control （2D-IPILC） strategy for batch processes is presented. First, the output response and the error transition model predictions along the batch index can be calculated analytically due to the 2D Roesser model of the batch process. Then, an integrated framework of combining iterative learning control （ILC） and model predictive control （MPC） is formed reasonably. The output of feedforward ILC is estimated on the basis of the predefined process 2D model. By min- imizing a quadratic objective function, the feedback MPC is introduced to obtain better control performance for tracking problem of batch processes. Simulations on a typical batch reactor demonstrate that the satisfactory tracking performance as well as faster convergence speed can be achieved than traditional proportion type （P- t-we） ILC despite the model error and disturbances.

Fuzzy NN based predictive control and its application to green liquor system

The fuzzy NN predictive control algorithm introduced in this paper uses fuzzy neural network to model the nonlinear MIMO process. Its training method that integrates LS and BP algorithm brings quick convergence. GPC algorithm is used as the predictive component. The fuzzy neural network has six layers, including input layer, output layer and four hidden layers. An application to a MIMO nonlinear process(green liquor system of the recovery system in a pulp factory shows that this algorithm has better performance than normal PID algrithm.

Multivariable Fuzzy Predictive Control Based on the Modified CPN Model

Through modifying the CPN model, a kind of multivariable fuzzy model is put forward, and the matching fuzzy multistep predictive control algorithm is deduced based on the model. The modified model works in a competitive output manner which results in its local representation property. While studying on line, only a few parameters need to be regulated. So the model has the merits of fast learning and on line self organizing modeling. The control algorithm is simple, adaptive and useful in multivariable and time delay systems. Applying the algorithm in a paper making system, simulation shows its good effect.

Safe operation of online learning data driven model predictive control of building energy systems

Model predictive control is a promising approach to reduce the CO 2 emissions in the building sector.However,the vast modeling effort hampers the widescale practical application.Here,data-driven process models,like artificial neural networks,are well-suited to automatize the modeling.However,the underlying data set strongly determines the quality and reliability of artificial neural networks.In general,the validity domain of a machine learning model is limited to the data that was used to train it.Predictions based on system states outside that domain,so-called extrapolations,are unreliable and can negatively influence the control quality.We present a safe operation approach combined with online learning to deal with extrapolation in data-driven model predictive control.Here,the k-nearest neighbor algorithm is used to detect extrapolation to switch to a robust fallback controller.By continuously retraining the artificial neural networks during operation,we successively increase the validity domain of the artificial neural networks and the control quality.We apply the approach to control a building energy system provided by the BOPTEST framework.We compare controllers based on two data sets,one with extensive system excitation and one with baseline operation.The system is controlled to a fixed temperature set point in baseline operation.Therefore,the artificial neural networks trained on this data set tend to extrapolate in other operating points.We show that safe operation in combination with online learning significantly improves performance.

Fuzzy iterative learning control of electro-hydraulic servo system for SRM direct-drive volume control hydraulic press

A new kind of volume control hydraulic press that combines the advantages of both hydraulic and SRM(switched reluctance motor) driving technology is developed.Considering that the serious dead zone and time-variant nonlinearity exist in the volume control electro-hydraulic servo system,the ILC(iterative learning control) method is applied to tracking the displacement curve of the hydraulic press slider.In order to improve the convergence speed and precision of ILC,a fuzzy ILC algorithm that utilizes the fuzzy strategy to adaptively adjust the iterative learning gains is put forward.The simulation and experimental researches are carried out to investigate the convergence speed and precision of the fuzzy ILC for hydraulic press slider position tracking.The results show that the fuzzy ILC can raise the iterative learning speed enormously,and realize the tracking control of slider displacement curve with rapid response speed and high control precision.In experiment,the maximum tracking error 0.02 V is achieved through 12 iterations only.

Constrained predictive control based on T-S fuzzy model for nonlinear systems

A constrained generalized predictive control （GPC） algorithm based on the T-S fuzzy model is presented for the nonlinear system. First, a Takagi-Sugeno （T-S） fuzzy model based on the fuzzy cluster algorithm and the orthogonalleast square method is constructed to approach the nonlinear system. Since its consequence is linear, it can divide the nonlinear system into a number of linear or nearly linear subsystems. For this T-S fuzzy model, a GPC algorithm with input constraints is presented. This strategy takes into account all the constraints of the control signal and its increment, and does not require the calculation of the Diophantine equations. So it needs only a small computer memory and the computational speed is high. The simulation results show a good performance for the nonlinear systems.

Robust model predictive control for discrete uncertain nonlinear systems with time-delay via fuzzy model

An extended robust model predictive control approach for input constrained discrete uncertain nonlinear systems with time-delay based on a class of uncertain T-S fuzzy models that satisfy sector bound condition is presented. In this approach, the minimization problem of the “worst-case” objective function is converted into the linear objective minimization problem in- volving linear matrix inequalities (LMIs) constraints. The state feedback control law is obtained by solving convex optimization of a set of LMIs. Sufficient condition for stability and a new upper bound on robust performance index are given for these kinds of uncertain fuzzy systems with state time-delay. Simulation results of CSTR process show that the proposed robust predictive control approach is effective and feasible.

Neuro-fuzzy generalized predictive control of boiler steam temperature

Power plants are nonlinear and uncertain complex systems. Reliable control of superheated steam temperature is necessary to ensure high efficiency and high load-following capability in the operation of modem power plant. A nonlinear generalized predictive controller based on neuro-fuzzy network （NFGPC） is proposed in this paper. The proposed nonlinear controller is applied to control the superheated steam temperature of a 200MW power plant. From the experiments on the plant and the simulation of the plant, much better performance than the traditional controller is obtained,

Fuzzy adaptive learning control network with sigmoid membership function

To get simpler operation in modified fuzzy adaptive learning control network （FALCON） in some engineering application, sigmoid nonlinear function is employed as a substitute of traditional Gaussian membership function. For making the modified FALCON learning more efficient and stable, a simulated annealing （SA） learning coefficient is introduced into learning algorithm. At first, the basic concepts and main advantages of FALCON were briefly reviewed. Subsequently, the topological structure and nodes operation were illustrated; the gradient-descent learning algorithm with SA learning coefficient was derived; and the distinctions between the archetype and the modification were analyzed. Eventually, the significance and worthiness of the modified FALCON were validated by its application to probability prediction of anode effect in aluminium electrolysis cells.

Neuro-fuzzy predictive control for nonlinear application

Aiming at the unsatisfactory dynamic performances of conventional model predictive control (MPC) in a highly nonlinear process, a scheme employed the fuzzy neural network to realize the nonlinear process is proposed. The neuro-fuzzy predictor has the capability of achieving high predictive accuracy due to its nonlinear mapping and interpolation features, and adaptively updating network parameters by a learning procedure to reduce the model errors caused by changes of the process under control. To cope with the difficult problem of nonlinear optimization, Pepanaqi method was applied to search the optimal or suboptimal solution. Comparisons were made among the objective function values of alternatives in initial space. The search was then confined to shrink the smaller region according to results of comparisons. The convergent point was finally approached to be considered as the optimal or suboptimal solution. Experimental results of the neuro-fuzzy predictive control for drier application reveal that the proposed control scheme has less tracking errors and can smooth control actions, which is applicable to changes of drying condition.

An Improved Fuzzy Predictive Control Algorithm and Its Application to an Industrial CSTR Process

A finite horizon predictive control algorithm, which applies a saturated feedback control law as its local control law, is presented for nonlinear systems with time-delay subject to input constraints. In the algorithm, N free control moves, a saturated local control law and the terminal weighting matrices are solved by a minimization problem based on linear matrix inequality （LMI） constraints online. Compared with the algorithm with a nonsaturated local law, the presented algorithm improves the performances of the closed-loop systems such as feasibility and optimality. This model predictive control （MPC） algorithm is applied to an industrial continuous stirred tank reactor （CSTR） with explicit input constraint. The simulation results demonstrate that the presented algorithm is effective.

2D multi-model general predictive iterative learning control for semi-batch reactor with multiple reactions

Batch to batch temperature control of a semi-batch chemical reactor with heating/cooling system was discussed in this study. Without extensive modeling investigations, a two-dimensional(2D) general predictive iterative learning control(2D-MGPILC) strategy based on the multi-model with time-varying weights was introduced for optimizing the tracking performance of desired temperature profile. This strategy was modeled based on an iterative learning control(ILC) algorithm for a 2D system and designed in the generalized predictive control(GPC) framework. Firstly, a multi-model structure with time-varying weights was developed to describe the complex operation of a general semi-batch reactor. Secondly, the 2 D-MGPILC algorithm was proposed to optimize simultaneously the dynamic performance along the time and batch axes. Finally, simulation for the controller design of a semi-batch reactor with multiple reactions was involved to demonstrate that the satisfactory performance could be achieved despite of the repetitive or non-repetitive disturbances.