Autonomous Vehicle Platoons In Urban Road Networks:A Joint Distributed Reinforcement Learning and Model Predictive Control Approach

In this paper, platoons of autonomous vehicles operating in urban road networks are considered. From a methodological point of view, the problem of interest consists of formally characterizing vehicle state trajectory tubes by means of routing decisions complying with traffic congestion criteria. To this end, a novel distributed control architecture is conceived by taking advantage of two methodologies: deep reinforcement learning and model predictive control. On one hand, the routing decisions are obtained by using a distributed reinforcement learning algorithm that exploits available traffic data at each road junction. On the other hand, a bank of model predictive controllers is in charge of computing the more adequate control action for each involved vehicle. Such tasks are here combined into a single framework:the deep reinforcement learning output(action) is translated into a set-point to be tracked by the model predictive controller;conversely, the current vehicle position, resulting from the application of the control move, is exploited by the deep reinforcement learning unit for improving its reliability. The main novelty of the proposed solution lies in its hybrid nature: on one hand it fully exploits deep reinforcement learning capabilities for decisionmaking purposes;on the other hand, time-varying hard constraints are always satisfied during the dynamical platoon evolution imposed by the computed routing decisions. To efficiently evaluate the performance of the proposed control architecture, a co-design procedure, involving the SUMO and MATLAB platforms, is implemented so that complex operating environments can be used, and the information coming from road maps(links,junctions, obstacles, semaphores, etc.) and vehicle state trajectories can be shared and exchanged. Finally by considering as operating scenario a real entire city block and a platoon of eleven vehicles described by double-integrator models, several simulations have been performed with the aim to put in light the main f eatures of the proposed approach. Moreover, it is important to underline that in different operating scenarios the proposed reinforcement learning scheme is capable of significantly reducing traffic congestion phenomena when compared with well-reputed competitors.

A Combined Reinforcement Learning and Model Predictive Control for Car-Following Maneuver of Autonomous Vehicles

Model predictive control is widely used in the design of autonomous driving algorithms.However,its parameters are sensitive to dynamically varying driving conditions,making it difficult to be implemented into practice.As a result,this study presents a self-learning algorithm based on reinforcement learning to tune a model predictive controller.Specifically,the proposed algorithm is used to extract features of dynamic traffic scenes and adjust the weight coefficients of the model predictive controller.In this method,a risk threshold model is proposed to classify the risk level of the scenes based on the scene features,and aid in the design of the reinforcement learning reward function and ultimately improve the adaptability of the model predictive controller to real-world scenarios.The proposed algorithm is compared to a pure model predictive controller in car-following case.According to the results,the proposed method enables autonomous vehicles to adjust the priority of performance indices reasonably in different scenarios according to risk variations,showing a good scenario adaptability with safety guaranteed.

Machine Learning Accelerated Real-Time Model Predictive Control for Power Systems

This paper presents a machine-learning-based speedup strategy for real-time implementation of model-predictive-control(MPC)in emergency voltage stabilization of power systems.Despite success in various applications,real-time implementation of MPC in power systems has not been successful due to the online control computation time required for large-sized complex systems,and in power systems,the computation time exceeds the available decision time used in practice by a large extent.This long-standing problem is addressed here by developing a novel MPC-based framework that i)computes an optimal strategy for nominal loads in an offline setting and adapts it for real-time scenarios by successive online control corrections at each control instant utilizing the latest measurements,and ii)employs a machine-learning based approach for the prediction of voltage trajectory and its sensitivity to control inputs,thereby accelerating the overall control computation by multiple times.Additionally,a realistic control coordination scheme among static var compensators(SVC),load-shedding(LS),and load tap-changers(LTC)is presented that incorporates the practical delayed actions of the LTCs.The performance of the proposed scheme is validated for IEEE 9-bus and 39-bus systems,with±20%variations in nominal loading conditions together with contingencies.We show that our proposed methodology speeds up the online computation by 20-fold,bringing it down to a practically feasible value(fraction of a second),making the MPC real-time and feasible for power system control for the first time.

Iterative Learning Model Predictive Control for a Class of Continuous/Batch Processes

An iterative learning model predictive control （ILMPC） technique is applied to a class of continuous/batch processes. Such processes are characterized by the operations of batch processes generating periodic strong disturbances to the continuous processes and traditional regulatory controllers are unable to eliminate these periodic disturbances. ILMPC integrates the feature of iterative learning control （ILC） handling repetitive signal and the flexibility of model predictive control （MPC）. By on-line monitoring the operation status of batch processes, an event-driven iterative learning algorithm for batch repetitive disturbances is initiated and the soft constraints are adjusted timely as the feasible region is away from the desired operating zone. The results of an industrial application show that the proposed ILMPC method is effective for a class of continuous/batch processes.

2D multi-model general predictive iterative learning control for semi-batch reactor with multiple reactions

Batch to batch temperature control of a semi-batch chemical reactor with heating/cooling system was discussed in this study. Without extensive modeling investigations, a two-dimensional(2D) general predictive iterative learning control(2D-MGPILC) strategy based on the multi-model with time-varying weights was introduced for optimizing the tracking performance of desired temperature profile. This strategy was modeled based on an iterative learning control(ILC) algorithm for a 2D system and designed in the generalized predictive control(GPC) framework. Firstly, a multi-model structure with time-varying weights was developed to describe the complex operation of a general semi-batch reactor. Secondly, the 2 D-MGPILC algorithm was proposed to optimize simultaneously the dynamic performance along the time and batch axes. Finally, simulation for the controller design of a semi-batch reactor with multiple reactions was involved to demonstrate that the satisfactory performance could be achieved despite of the repetitive or non-repetitive disturbances.

Optimal Iterative Learning Control for Batch Processes Based on Linear Time-varying Perturbation Model

A batch-to-batch optimal iterative learning control （ILC） strategy for the tracking control of product quality in batch processes is presented. The linear time-varying perturbation （LTVP） model is built for product quality around the nominal trajectories. To address problems of model-plant mismatches, model prediction errors in the previous batch run are added to the model predictions for the current batch run. Then tracking error transition models can be built, and the ILC law with direct error feedback is explicitly obtained, A rigorous theorem is proposed, to prove the convergence of tracking error under ILC, The proposed methodology is illustrated on a typical batch reactor and the results show that the performance of trajectory tracking is gradually improved by the ILC.

Design and Analysis of Integrated Predictive Iterative Learning Control for Batch Process Based on Two-dimensional System Theory

Based on the two-dimensional （2D） system theory, an integrated predictive iterative learning control （2D-IPILC） strategy for batch processes is presented. First, the output response and the error transition model predictions along the batch index can be calculated analytically due to the 2D Roesser model of the batch process. Then, an integrated framework of combining iterative learning control （ILC） and model predictive control （MPC） is formed reasonably. The output of feedforward ILC is estimated on the basis of the predefined process 2D model. By min- imizing a quadratic objective function, the feedback MPC is introduced to obtain better control performance for tracking problem of batch processes. Simulations on a typical batch reactor demonstrate that the satisfactory tracking performance as well as faster convergence speed can be achieved than traditional proportion type （P- t-we） ILC despite the model error and disturbances.

Multi-loop Constrained Iterative Model Predictive Control Using ARX -PLS Decoupling Structure

A multi-loop constrained model predictive control scheme based on autoregressive exogenous-partial least squares(ARX-PLS) framework is proposed to tackle the high dimension, coupled and constraints problems in industry processes due to safety limitation, environmental regulations, consumer specifications and physical restriction. ARX-PLS decoupling character enables to turn the multivariable model predictive control(MPC) controller design in original space into the multi-loop single input single output(SISO) MPC controllers design in latent space.An idea of iterative method is applied to decouple the constraints latent variables in PLS framework and recursive least square is introduced to identify ARX-PLS model. This algorithm is applied to a non-square simulation system and a stirred reactor for ethylene polymerizations comparing with adaptive internal model control(IMC) method based on ARX-PLS framework. Its application has shown that this method outperforms adaptive IMC method based on ARX-PLS framework to some extent.

An LMI Method to Robust Iterative Learning Fault-tolerant Guaranteed Cost Control for Batch Processes

Based on an equivalent two-dimensional Fornasini-Marchsini model for a batch process in industry, a closed-loop robust iterative learning fault-tolerant guaranteed cost control scheme is proposed for batch processes with actuator failures. This paper introduces relevant concepts of the fault-tolerant guaranteed cost control and formulates the robust iterative learning reliable guaranteed cost controller (ILRGCC). A significant advantage is that the proposed ILRGCC design method can be used for on-line optimization against batch-to-batch process uncertainties to realize robust tracking of set-point trajectory in time and batch-to-batch sequences. For the convenience of implementation, only measured output errors of current and previous cycles are used to design a synthetic controller for iterative learning control, consisting of dynamic output feedback plus feed-forward control. The proposed controller can not only guarantee the closed-loop convergency along time and cycle sequences but also satisfy the H∞performance level and a cost function with upper bounds for all admissible uncertainties and any actuator failures. Sufficient conditions for the controller solution are derived in terms of linear matrix inequalities (LMIs), and design procedures, which formulate a convex optimization problem with LMI constraints, are presented. An example of injection molding is given to illustrate the effectiveness and advantages of the ILRGCC design approach.

Distributed Model Predictive Control with Actuator Saturation for Markovian Jump Linear System

This paper is concerned with the distributed model predictive control (MPC) problem for a class of discrete-time Markovian jump linear systems (MJLSs) subject to actuator saturation and polytopic uncertainty in system matrices. The global system is decomposed into several subsystems which coordinate with each other. A set of distributed controllers is designed by solving a min-max optimization problem in terms of the solutions of linear matrix inequalities (LMIs). An iterative algorithm is developed to achieve the online computation. Finally, a simulation example is employed to show the effectiveness of the proposed algorithm. © 2014 Chinese Association of Automation.

Dual-stage Optimal Iterative Learning Control for Nonlinear Non-affine Discrete-time Systems

根据沿着重复轴的一种新动态 linearization 技术，双阶段的最佳的反复的学习控制为非线性、非仿射的分离时间的系统被介绍。双阶段显示二个最佳的学习阶段分别地被设计反复地改进控制输入顺序和学习获得。主要特征是控制器设计和集中分析仅仅取决于动态系统的 I/O 数据。换句话说，没有知道系统的任何另外的知识，我们能容易选择控制参数。模拟学习沿着重复轴说明介绍方法的几何集中，在哪个马路的一个例子控制为它的内在的工程重要性是引人注目的交通反复的学习。

A Simulation Study of Higee Rotor Auto-balancing Based on Iterative Learning Control

Hypergravity technology has a wide application prospect on many industry areas for its powerful ability on multiphase flow transport and reaction.In its long-term operation,vibration control of higee rotor is an important guarantee for high-quality continuous outputs.Offline approach has great influence on continuity of the whole production line.In order to study online auto-balancing control strategy,a mathematical model of higee rotor was established.Then basic Iterative Learning Control(ILC)algorithm and its improved structure based on vector analysis were introduced.Pure injection balancer and electromagnetic balancer were separately used as the actuator.Three different control algorithms(P control using Cohen-Coon parameter tuning law,basic ILC,and improved ILC based on vector analysis)were compared under single eccentric mass disturbance and continuous ones.Simulation results manifested the effects of ILC in rotor auto-balancing control,especially on the "over-control" issue during the balancing process.

A Novel Tuning Method for Predictive Control of VAV Air Conditioning System Based on Machine Learning and Improved PSO

The variable air volume(VAV)air conditioning system is with strong coupling and large time delay,for which model predictive control(MPC)is normally used to pursue performance improvement.Aiming at the difficulty of the parameter selection of VAV MPC controller which is difficult to make the system have a desired response,a novel tuning method based on machine learning and improved particle swarm optimization(PSO)is proposed.In this method,the relationship between MPC controller parameters and time domain performance indices is established via machine learning.Then the PSO is used to optimize MPC controller parameters to get better performance in terms of time domain indices.In addition,the PSO algorithm is further modified under the principle of population attenuation and event triggering to tune parameters of MPC and reduce the computation time of tuning method.Finally,the effectiveness of the proposed method is validated via a hardware-in-the-loop VAV system.

Just-in-time learning based integrated MPC-ILC control for batch processes

Considering the two-dimension(2 D) characteristic and the unknown optimal trajectory problem of the batch processes, an integrated model predictive control-iterative learning control(MPC-ILC) for batch processes is proposed in this paper. Firstly, the batch-axis information and time-axis information are combined into one quadratic performance index. It implies the integration of ILC and MPC algorithm idea, which leads to superior tracking performance and better robustness against disturbance and uncertainty. To address the problem of the unknown optimal trajectory, both time-varying prediction horizon and end product quality control are employed. Moreover, an integrated 2 D just-in-time learning(JITL) model is used to improve the predictive accuracy. Furthermore, rigorous description and proof are presented to prove the convergence and tracking performance of the proposed MPC-ILC strategy. The simulation results show the effectiveness of the proposed method.

Real-Time Iterative Compensation Framework for Precision Mechatronic Motion Control Systems

With regard to precision/ultra-precision motion systems,it is important to achieve excellent tracking performance for various trajectory tracking tasks even under uncertain external disturbances.In this paper,to overcome the limitation of robustness to trajectory variations and external disturbances in offline feedforward compensation strategies such as iterative learning control(ILC),a novel real-time iterative compensation(RIC)control framework is proposed for precision motion systems without changing the inner closed-loop controller.Specifically,the RIC method can be divided into two parts,i.e.,accurate model prediction and real-time iterative compensation.An accurate prediction model considering lumped disturbances is firstly established to predict tracking errors at future sampling times.In light of predicted errors,a feedforward compensation term is developed to modify the following reference trajectory by real-time iterative calculation.Both the prediction and compen-sation processes are finished in a real-time motion control sampling period.The stability and convergence of the entire control system after real-time iterative compensation is analyzed for different conditions.Various simulation results consistently demonstrate that the proposed RIC framework possesses satisfactory dynamic regulation capability,which contributes to high tracking accuracy comparable to ILC or even better and strong robustness.

Batch Process Modelling and Optimal Control Based on Neural Network Model

This paper presents several neural network based modelling, reliable optimal control, and iterative learning control methods for batch processes. In order to overcome the lack of robustness of a single neural network, bootstrap aggregated neural networks are used to build reliable data based empirical models. Apart from improving the model generalisation capability, a bootstrap aggregated neural network can also provide model prediction confidence bounds. A reliable optimal control method by incorporating model prediction confidence bounds into the optimisation objective function is presented. A neural network based iterative learning control strategy is presented to overcome the problem due to unknown disturbances and model-plant mismatches. The proposed methods are demonstrated on a simulated batch polymerisation process.

A Variable-Parameter-Model-Based Feedforward Compensation Method for Tracking Control

Base on the accurate inverse of a system, the feedforward compensation method can compensate the tracking error of a linear system dramatically. However, many control systems have complex dynamics and their accurate inverses are difficult to obtain. In the paper, a variable parameter model is proposed to describe a system and a multi-step adaptive seeking approach is used to obtain its parameters in real time. Based on the proposed model, a variable-parameter-model-based feedforward compensation method is proposed, and a disturbance observer is used to overcome the influence of the model uncertainty. Theoretical analysis and simulation results show that the variable-parametermodel-based feedforward compensation method can obtain better performance than the traditional feedforward compensation.

Safe operation of online learning data driven model predictive control of building energy systems

Model predictive control is a promising approach to reduce the CO 2 emissions in the building sector.However,the vast modeling effort hampers the widescale practical application.Here,data-driven process models,like artificial neural networks,are well-suited to automatize the modeling.However,the underlying data set strongly determines the quality and reliability of artificial neural networks.In general,the validity domain of a machine learning model is limited to the data that was used to train it.Predictions based on system states outside that domain,so-called extrapolations,are unreliable and can negatively influence the control quality.We present a safe operation approach combined with online learning to deal with extrapolation in data-driven model predictive control.Here,the k-nearest neighbor algorithm is used to detect extrapolation to switch to a robust fallback controller.By continuously retraining the artificial neural networks during operation,we successively increase the validity domain of the artificial neural networks and the control quality.We apply the approach to control a building energy system provided by the BOPTEST framework.We compare controllers based on two data sets,one with extensive system excitation and one with baseline operation.The system is controlled to a fixed temperature set point in baseline operation.Therefore,the artificial neural networks trained on this data set tend to extrapolate in other operating points.We show that safe operation in combination with online learning significantly improves performance.

机器学习驱动锅炉燃烧优化技术的现状与展望

伴随可再生能源发电装机容量快速增加,深度调峰过程中负荷多变、燃烧失稳等不稳定工况对火电机组的燃烧优化控制提出了更高要求,快速发展的人工智能技术与深度学习算法为锅炉参数预测建模及优化提供了重要手段。在机器学习算法方面,总结了特征筛选与建模算法的研究现状,提出了传统统计学方法与线性降维方法的科学解释性较差且不能很好地辨识高维数据,结合深度学习算法的特征筛选方法在处理复杂的火电机组数据时优势更明显;对比了多种神经网络在NO_(x)排放浓度建模中的优缺点,其中长短期记忆神经网络与卷积神经网络在处理时序数据时效果更好、集成模型通过组合不同学习器的优势可提高整个模型的泛化能力和鲁棒性。在预测模型的应用方面,通过对SCR脱硝系统建立预测模型可以方便运行人员模拟并修正可调参数,同时作为软测量手段监测燃烧系统运行状态;引入NO_(x)排放浓度预测模型的前馈控制和模型预测控制等先进控制手段可有效改善火电机组传统PID控制效果较差的问题;在多目标优化中NO_(x)脱除效率通常与锅炉效率或脱硝成本共同作为优化目标,以期实现经济效益与社会效益的和谐统一。

基于KL散度工况识别的混合动力汽车队列的分层控制

针对混合动力汽车队列行驶过程中工况的适应性问题,提出了一种基于KL(Kullback-Leibler)散度工况识别的分层控制方法。上层控制器利用车—车通信技术,获取队列中前车状态信息,采用模型预测控制(MPC)算法,实现队列纵向控制,并计算出最优跟车车速;下层控制器基于典型工况,离线求解需求功率的转移概率矩阵,并通过Q-Learning算法训练最优Q表嵌入整车模型中;在行驶中以固定时间长度在线更新转移概率矩阵,采用KL散度进行工况识别,根据识别的工况类型,结合当前时刻车速、需求功率和电池荷电状态(SOC),通过在线查表实现转矩分配。结果表明:与未考虑工况识别策略相比,本策略的油耗降低了8.6%;与作为基准的动态规划(DP)相比,增加了4.8%;在与DP油耗基本保持相同的前提下,该策略离线仿真时间缩短21%,不仅能够在线应用,还能实时适应工况变化。