Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Adaptive Multi-Step Evaluation Design With Stability Guarantee for Discrete-Time Optimal Learning Control

This paper is concerned with a novel integrated multi-step heuristic dynamic programming(MsHDP)algorithm for solving optimal control problems.It is shown that,initialized by the zero cost function,MsHDP can converge to the optimal solution of the Hamilton-Jacobi-Bellman(HJB)equation.Then,the stability of the system is analyzed using control policies generated by MsHDP.Also,a general stability criterion is designed to determine the admissibility of the current control policy.That is,the criterion is applicable not only to traditional value iteration and policy iteration but also to MsHDP.Further,based on the convergence and the stability criterion,the integrated MsHDP algorithm using immature control policies is developed to accelerate learning efficiency greatly.Besides,actor-critic is utilized to implement the integrated MsHDP scheme,where neural networks are used to evaluate and improve the iterative policy as the parameter architecture.Finally,two simulation examples are given to demonstrate that the learning effectiveness of the integrated MsHDP scheme surpasses those of other fixed or integrated methods.

Adaptive learning tracking control of robotic manipulators with uncertainties

An adaptive learning tracking control scheme is developed for robotic manipulators by a synthesis of adaptive control and learning control approaches. The proposed controller possesses both adaptive and learning properties and thereby is able to handle robotic systems with both time-varying periodic uncertainties and time invariant parameters. Theoretical proofs are established to show that proposed controllers ensure asymptotical tracking performance. The effectiveness of the proposed approaches is validated through extensive numerical simulation results.

Mobility-Aware Adaptive Beam Tracking for Vehicles in Mm Wave Communication Networks

The millimeter wave(mm Wave)is a potential solution for high data rate communication due to its availability of large bandwidth.However,it is challenging to perform beam tracking in vehicular mm Wave communication systems due to high mobility and narrow beams.In this paper,an adaptive beam tracking algorithm is proposed to improve the network throughput performance while reducing the training signal overhead.In particular,based on the mobility prediction at base station(BS),a novel frame structure with dynamic bundled timeslot is designed.Moreover,an actor-critic reinforcement learning based algorithm is proposed to obtain the joint optimization of both beam width and the number of bundled timeslots,which makes the beam tracking adapt to the changing environment.Simulation results demonstrate that,compared with the traditional full scan and Kalman filter based beam tracking algorithms,our proposed algorithm can improve the time-averaged throughput by 11.34%and 24.86%respectively.With the newly designed frame structure,it also outperforms beam tracking with conventional frame structure,especially in scenarios with large range of vehicle speeds.

Self-adaptive large neighborhood search algorithm for parallel machine scheduling problems

A self-adaptive large neighborhood search method for scheduling n jobs on m non-identical parallel machines with mul- tiple time windows is presented. The problems＇ another feature lies in oversubscription, namely not all jobs can be scheduled within specified scheduling horizons due to the limited machine capacity. The objective is thus to maximize the overall profits of processed jobs while respecting machine constraints. A first-in- first-out heuristic is applied to find an initial solution, and then a large neighborhood search procedure is employed to relax and re- optimize cumbersome solutions. A machine learning mechanism is also introduced to converge on the most efficient neighborhoods for the problem. Extensive computational results are presented based on data from an application involving the daily observation scheduling of a fleet of earth observing satellites. The method rapidly solves most problem instances to optimal or near optimal and shows a robust performance in sensitive analysis.

Dynamic Distribution Adaptation Based Transfer Network for Cross Domain Bearing Fault Diagnosis

In machinery fault diagnosis,labeled data are always difficult or even impossible to obtain.Transfer learning can leverage related fault diagnosis knowledge from fully labeled source domain to enhance the fault diagnosis performance in sparsely labeled or unlabeled target domain,which has been widely used for cross domain fault diagnosis.However,existing methods focus on either marginal distribution adaptation(MDA)or conditional distribution adaptation(CDA).In practice,marginal and conditional distributions discrepancies both have significant but different influences on the domain divergence.In this paper,a dynamic distribution adaptation based transfer network(DDATN)is proposed for cross domain bearing fault diagnosis.DDATN utilizes the proposed instance-weighted dynamic maximum mean discrepancy(IDMMD)for dynamic distribution adaptation(DDA),which can dynamically estimate the influences of marginal and conditional distribution and adapt target domain with source domain.The experimental evaluation on cross domain bearing fault diagnosis demonstrates that DDATN can outperformance the state-of-the-art cross domain fault diagnosis methods.

Dynamic Intelligent Supply-Demand Adaptation Model Towards Intelligent Cloud Manufacturing

As a new mode and means of smart manufacturing,smart cloud manufacturing(SCM)faces great challenges in massive supply and demand,dynamic resource collaboration and intelligent adaptation.To address the problem,this paper proposes an SCM-oriented dynamic supply-demand(SD)intelligent adaptation model for massive manufacturing services.In this model,a collaborative network model is established based on the properties of both the supply-demand and their relationships;in addition,an algorithm based on deep graph clustering(DGC)and aligned sampling(AS)is used to divide and conquer the large adaptation domain to solve the problem of the slow computational speed caused by the high complexity of spatiotemporal search in the collaborative network model.At the same time,an intelligent supply-demand adaptation method driven by the quality of service(QoS)is established,in which the experiences of adaptation are shared among adaptation subdomains through deep reinforcement learning(DRL)powered by a transfer mechanism to improve the poor adaptation results caused by dynamic uncertainty.The results show that the model and the solution proposed in this paper can performcollaborative and intelligent supply-demand adaptation for themassive and dynamic resources in SCM through autonomous learning and can effectively performglobal supply-demand matching and optimal resource allocation.

PDP: Parallel Dynamic Programming

Deep reinforcement learning is a focus research area in artificial intelligence. The principle of optimality in dynamic programming is a key to the success of reinforcement learning methods. The principle of adaptive dynamic programming(ADP)is first presented instead of direct dynamic programming(DP),and the inherent relationship between ADP and deep reinforcement learning is developed. Next, analytics intelligence, as the necessary requirement, for the real reinforcement learning, is discussed. Finally, the principle of the parallel dynamic programming, which integrates dynamic programming and analytics intelligence, is presented as the future computational intelligence.

Advanced Policy Learning Near-Optimal Regulation

Designing advanced design techniques for feedback stabilization and optimization of complex systems is important to the modern control field. In this paper, a near-optimal regulation method for general nonaffine dynamics is developed with the help of policy learning. For addressing the nonaffine nonlinearity, a pre-compensator is constructed, so that the augmented system can be formulated as affine-like form. Different cost functions are defined for original and transformed controlled plants and then their relationship is analyzed in detail. Additionally, an adaptive critic algorithm involving stability guarantee is employed to solve the augmented optimal control problem. At last, several case studies are conducted for verifying the stability, robustness, and optimality of a torsional pendulum plant with suitable cost.

Discounted Iterative Adaptive Critic Designs With Novel Stability Analysis for Tracking Control

The core task of tracking control is to make the controlled plant track a desired trajectory.The traditional performance index used in previous studies cannot eliminate completely the tracking error as the number of time steps increases.In this paper,a new cost function is introduced to develop the value-iteration-based adaptive critic framework to solve the tracking control problem.Unlike the regulator problem,the iterative value function of tracking control problem cannot be regarded as a Lyapunov function.A novel stability analysis method is developed to guarantee that the tracking error converges to zero.The discounted iterative scheme under the new cost function for the special case of linear systems is elaborated.Finally,the tracking performance of the present scheme is demonstrated by numerical results and compared with those of the traditional approaches.

Online Learning Control for Harmonics Reduction Based on Current Controlled Voltage Source Power Inverters

Nonlinear loads in the power distribution system cause non-sinusoidal currents and voltages with harmonic components.Shunt active filters(SAF) with current controlled voltage source inverters(CCVSI) are usually used to obtain balanced and sinusoidal source currents by injecting compensation currents.However,CCVSI with traditional controllers have a limited transient and steady state performance.In this paper,we propose an adaptive dynamic programming(ADP) controller with online learning capability to improve transient response and harmonics.The proposed controller works alongside existing proportional integral(PI) controllers to efficiently track the reference currents in the d-q domain.It can generate adaptive control actions to compensate the PI controller.The proposed system was simulated under different nonlinear(three-phase full wave rectifier) load conditions.The performance of the proposed approach was compared with the traditional approach.We have also included the simulation results without connecting the traditional PI control based power inverter for reference comparison.The online learning based ADP controller not only reduced average total harmonic distortion by 18.41%,but also outperformed traditional PI controllers during transients.

Consensus control for heterogeneous uncertain multi-agent systems with hybrid nonlinear dynamics via iterative learning algorithm

In this study,We propose a compensated distributed adaptive learning algorithm for heterogeneous multi-agent systems with repetitive motion,where the leader's dynamics are unknown,and the controlled system's parameters are uncertain.The multiagent systems are considered a kind of hybrid order nonlinear systems,which relaxes the strict requirement that all agents are of the same order in some existing work.For theoretical analyses,we design a composite energy function with virtual gain parameters to reduce the restriction that the controller gain depends on global information.Considering the stability of the controller,we introduce a smooth continuous function to improve the piecewise controller to avoid possible chattering.Theoretical analyses prove the convergence of the presented algorithm,and simulation experiments verify the effectiveness of the algorithm.

Dynamic Movement Primitives Based Robot Skills Learning

In this article,a robot skills learning framework is developed,which considers both motion modeling and execution.In order to enable the robot to learn skills from demonstrations,a learning method called dynamic movement primitives(DMPs)is introduced to model motion.A staged teaching strategy is integrated into DMPs frameworks to enhance the generality such that the complicated tasks can be also performed for multi-joint manipulators.The DMP connection method is used to make an accurate and smooth transition in position and velocity space to connect complex motion sequences.In addition,motions are categorized into different goals and durations.It is worth mentioning that an adaptive neural networks(NNs)control method is proposed to achieve highly accurate trajectory tracking and to ensure the performance of action execution,which is beneficial to the improvement of reliability of the skills learning system.The experiment test on the Baxter robot verifies the effectiveness of the proposed method.

State of the Art of Adaptive Dynamic Programming and Reinforcement Learning

This article introduces the state-of-the-art development of adaptive dynamic programming and reinforcement learning(ADPRL).First,algorithms in reinforcement learning(RL)are introduced and their roots in dynamic programming are illustrated.Adaptive dynamic programming(ADP)is then introduced following a brief discussion of dynamic programming.Researchers in ADP and RL have enjoyed the fast developments of the past decade from algorithms,to convergence and optimality analyses,and to stability results.Several key steps in the recent theoretical developments of ADPRL are mentioned with some future perspectives.In particular,convergence and optimality results of value iteration and policy iteration are reviewed,followed by an introduction to the most recent results on stability analysis of value iteration algorithms.

多场景下基于传感器的行为识别

针对基于传感器的行为识别任务中识别场景单一且固定的问题,提出一种多场景下基于传感器的行为识别迁移模型,由基于传感器的动态感知算法(dynamic perception algorithm,DPA)和自适应场景的行为识别迁移方法(adaptive scene human recognition,AHR)两部分组成,解决在固定场景下对传感器的依赖性以及在场景转换时识别模型失效的问题。DPA提出两阶段迁移模式,将行为识别阶段和模型迁移阶段同步推进,保证模型在传感器异动发生后仍能持续拥有识别能力。进一步提出AHR场景迁移方法,实现模型在多场景下的行为识别能力。实验验证该模型具有更优的适应性和可扩展性。

从管段走向管网:管道泄漏诊断技术研究进展

管道泄漏诊断技术在保障管道系统安全运行中起着至关重要的作用。首先,介绍了管道泄漏诊断系统的结构,并指出由单一管段向复杂管网泄漏诊断的发展趋势。进一步从基于数据驱动的传统泄漏检测方法、管道泄漏信号源定位技术和基于深度学习的复杂管网泄漏检测方法3个方面进行综述,分析了不同方法的优势、局限性和适用范围。最后指出,随着管网系统的复杂度增加,传统方法的局限性逐渐显现,基于深度学习技术的复杂管网微弱泄漏诊断、多源信号融合和管网智能化的研究将成为未来的研究趋势。

一种基于深度自适应网络迁移的暂稳评估模型更新框架

为解决电力系统的运行方式或拓扑结构变化后暂稳评估模型的适应性问题,常规的特征迁移学习方法主要侧重于拉近源域与目标域数据集间的条件分布或边缘分布的距离,却不能定量的评价这两种分布对于不同域之间的贡献,导致模型迁移性能不理想。针对该问题,引入SENet注意力机制和动态分布自适应算法,构建了基于SEDDAN迁移的深度自适应网络暂稳评估模型更新框架,从特征提取和不同域间分布权重的动态调整两个层面进行改进,进一步提升了评估模型的迁移性能和自适应性。在IEEE 39和IEEE 140节点系统上进行测试,仿真结果表明所提模型在更新后的评估准确性、适应性和迁移性能方面有一定的优势。

基于新型细菌觅食优化算法的飞机动态泊位问题

随着航空运输业的发展,传统手动设计泊位方案已难以满足日益增长的外包维修需求.在外包模式下,如何快速给出高效的动态泊位方案关系到维修任务订单的准点交付,是飞机维修服务公司亟待解决的重要问题.针对飞机泊位进出顺序及碰撞检测特点,构建带时间窗的飞机维修泊位模型.设计自适应趋化学习及交叉协作策略,提出新型细菌觅食优化算法,并设计一系列约束处理机制.研究结果表明,提出的基于矩形碰撞检测方法可有效预防并判断飞机间碰撞阻塞情况.新型细菌觅食优化算法在解决飞机动态泊位问题上展现出搜索精度高、稳定性强等特点.所得高效智能化泊位调度方案有助于在保证维修安全的情况下提升飞机维修服务提供商的维修服务效率,改进维修资源利用率与维修系统的柔性,为企业实现高质量发展打下良好基础.

多策略融合改进的自适应蜉蝣算法

为改进蜉蝣算法全局搜索能力较差、种群多样性较小和自适应能力弱等问题,提出一种多策略融合改进的自适应蜉蝣算法(MIMA)。采用Sin混沌映射初始化蜉蝣种群,使种群能够均匀分布在解空间中,提高初始种群质量,增强全局搜索能力;引入Tent混沌映射和高斯变异对种群个体进行调节,增加种群多样性的同时调控种群密度,增强局部最优逃逸能力;引入不完全伽马函数,重构自适应动态调节的重力系数,建立全局搜索和局部开发能力之间更好的平衡,进而提升算法收敛精度,有利于提高全局搜索能力;采用随机反向学习(ROBL)策略,增强全局搜索能力,提高收敛速度并增强稳定性。利用经典测试函数集进行算法对比,并利用Wilcoxon秩和检验分析算法的优化效果,证明改进的有效性和可靠性。实验结果表明:所提算法与其他算法相比,寻优精度、收敛速度、稳定性都取得了较大提升。

用于训练神经网络的自适应梯度下降优化算法

由于神经网络规模的扩大,模型训练变得越来越困难.为应对这一问题,提出了一种新的自适应优化算法——Adaboundinject.选取Adam的改进算法Adabound算法,引入动态学习率边界,实现了自适应算法向随机梯度下降(SGD)的平稳过渡.为了避免最小值的超调,减少在最小值附近的振荡,在Adabound的二阶矩中加入一阶矩,利用短期参数更新作为权重,以控制参数更新.为了验证算法性能,在凸环境下,通过理论证明了Adaboundinject具有收敛性.在非凸环境下,进行了多组实验,采用了不同的神经网络模型,通过与其他自适应算法对比,验证了该算法相比其他优化算法具有更好的性能.实验结果表明,Adaboundinject算法在深度学习优化领域具有重要的应用价值,能够有效提高模型训练的效率和精度.