Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Complex adaptive system theory,agent-based modeling,and simulation in dominant technology formation

Dominant technology formation is the key for the hightech industry to“cross the chasm”and gain an established foothold in the market(and hence disrupt the regime).Therefore,a stimulus-response model is proposed to investigate the dominant technology by exploring its formation process and mechanism.Specifically,based on complex adaptive system theory and the basic stimulus-response model,we use a combination of agent-based modeling and system dynamics modeling to capture the interactions between dominant technology and the socio-technical landscape.The results indicate the following:(i)The dynamic interaction is“stimulus-reaction-selection”,which promotes the dominant technology’s formation.(ii)The dominant technology’s formation can be described as a dynamic process in which the adaptation intensity of technology standards increases continuously until it becomes the leading technology under the dual action of internal and external mechanisms.(iii)The dominant technology’s formation in the high-tech industry is influenced by learning ability,the number of adopting users and adaptability.Therein,a“critical scale”of learning ability exists to promote the formation of leading technology:a large number of adopting users can promote the dominant technology’s formation by influencing the adaptive response of technology standards to the socio-technical landscape and the choice of technology standards by the socio-technical landscape.There is a minimum threshold and a maximum threshold for the role of adaptability in the dominant technology’s formation.(iv)The socio-technical landscape can promote the leading technology’s shaping in the high-tech industry,and different elements have different effects.This study promotes research on the formation mechanism of dominant technology in the high-tech industry,presents new perspectives and methods for researchers,and provides essential enlightenment for managers to formulate technology strategies.

Model for cascading failures with adaptive defense in complex networks

This paper investigates cascading failures in networks by considering interplay between the flow dynamic and the network topology, where the fluxes exchanged between a pair of nodes can be adaptively adjusted depending on the changes of the shortest path lengths between them. The simulations on both an artificially created scale-free network and the real network structure of the power grid reveal that the adaptive adjustment of the fluxes can drastically enhance the robustness of complex networks against cascading failures. Particularly, there exists an optimal region where the propagation of the cascade is significantly suppressed and the fluxes supported by the network are maximal. With this understanding, a costless strategy of defense for preventing cascade breakdown is proposed. It is shown to be more effective for suppressing the propagation of the cascade than the recent proposed strategy of defense based on the intentional removal of nodes.

基于CAS理论的非遗传统技艺传承模式评价指标体系设计

本文旨在探讨非遗传统技艺的传承模式受物理场景的局限以及传承主体的固化,导致传统技艺传承体验受限和传播目标缺失,已难适应新时代的需求现状。通过以复杂适应系统(Complex Adaptive System,CAS)理论为指导,借助“自适应主体”的引入,从“主体”与“环境”交互的角度审视复杂系统的行为特征,设计一个“体验+传播”的非物质文化遗产(非遗)传统技艺传承模式评价体系,为非遗保护、传承及其可持续发展提供更为科学、全面和动态的评价工具和策略,有助于实现非遗的活态保护和创造性转化。

基于复杂适应系统(CAS)理论的编辑出版人才培养机制研究

全面考察编辑出版人才培养中的各个主体及其所处的外部环境,探索新时代下编辑出版人才培养机制。文章引入复杂适应系统理论,明确系统内部的各个主体,并梳理微观层面上的主体适应性和宏观层面上的多元主体互动。依循核心主体—其他多元主体—主体与主体之间—主体与外部环境之间的各个系统层次,立足于对编辑出版人才主体地位的明确,提出新时代下政府、高校、出版单位、行业协会多主体联合的编辑出版人才培养机制。提出应尊重编辑出版人才主体及其个人发展,激发主体活力以释放新动能,促进主体间交互以实现整体涌现,并以多样化的人才培养成就多元化的编辑出版人才,以助力出版业的高质量发展。

Patterns of Interactions of the Complex City System:Emotional Urban Objects as Triggering Agents-A Secondary Publication

This article presents an analysis of the patterns of interactions resulting from the positive and negative emotional events that occur in cities,considering them as complex systems.It explores,from the imaginaries,how certain urban objects can act as emotional agents and how these events affect the urban system as a whole.An adaptive complex systems perspective is used to analyze these patterns.The results show patterns in the processes and dynamics that occur in cities based on the objects that affect the emotions of the people who live there.These patterns depend on the characteristics of the emotional charge of urban objects,but they can be generalized in the following process:(1)immediate reaction by some individuals;(2)emotions are generated at the individual level which begins to generalize,permuting to a collective emotion;(3)a process of reflection is detonated in some individuals from the reading of collective emotions;(4)integration/significance in the community both at the individual and collective level,on the concepts,roles and/or functions that give rise to the process in the system.Therefore,it is clear that emotions play a significant role in the development of cities and these aspects should be considered in the design strategies of all kinds of projects for the city.Future extensions of this work could include a deeper analysis of specific emotional events in urban environments,as well as possible implications for urban policy and decision making.

A Method for Reducing Ocean Wave-Induced Magnetic Noises in Shallow-Water MT Data Using a Complex Adaptive Filter

In shallow-water areas,the marine magnetotelluric(MT)method faces a challenge in the investigation of seabed conductivity structures due to electrical and magnetic noises induced by ocean waves,which seriously contaminate MT data.Ocean waves can affect electric and magnetic fields to different extents.In general,their influence on magnetic fields is considerably greater than that on electric fields.In this paper,a complex adaptive filter is adopted to reduce wave-induced magnetic noises in the frequency domain.The processing results of synthetic and measured MT data indicate that the proposed method can effectively reduce wave-induced magnetic noises and provide reliable apparent resistivity and phase data.

Adaptive generalized matrix projective lag synchronization between two different complex networks with non-identical nodes and different dimensions

The adaptive generalized matrix projective lag synchronization between two different complex networks with non-identical nodes and different dimensions is investigated in this paper. Based on Lyapunov stability theory and Barbalat＇s lemma, generalized matrix projective lag synchronization criteria are derived by using the adaptive control method. Furthermore, each network can be undirected or directed, connected or disconnected, and nodes in either network may have identical or different dynamics. The proposed strategy is applicable to almost all kinds of complex networks. In addition, numerical simulation results are presented to illustrate the effectiveness of this method, showing that the synchronization speed is sensitively influenced by the adaptive law strength, the network size, and the network topological structure.

Robust adaptive synchronization of uncertain and delayed dynamical complex networks with faulty network

This paper presents a new robust adaptive synchronization method for a class of uncertain dynamical complex networks with network failures and coupling time-varying delays. Adaptive schemes are proposed to adjust controller parameters for the faulty network compensations, as well as to estimate the upper and lower bounds of delayed state errors and perturbations to compensate the effects of delay and perturbation on-line without assuming symmetry or irreducibility of networks. It is shown that, through Lyapunov stability theory, distributed adaptive controllers con- structed by the adaptive schemes are successful in ensuring the achievement of asymptotic synchronization of networks in the present of faulty and delayed networks, and perturbation inputs. A Chua＇s circuit network example is finally given to show the effectiveness of the proposed synchronization criteria.

Adaptive function projective synchronization of uncertain complex dynamical networks with disturbance

We investigate the problem of function projective synchronization （FPS） in drive–response dynamical networks with non-identical nodes. An adaptive controller is proposed for the FPS of complex dynamical networks with uncertain parameters and disturbance. Not only are the unknown parameters of the networks estimated by the adaptive laws obtained from the Lyapunov stability theory and Taylor expansions, but the unknown bounded disturbances are also simultaneously conquered by the proposed control. Finally, a numerical simulation is provided to illustrate the feasibility and effectiveness of the obtained result.

Decentralized adaptive synchronization of an uncertain complex delayed dynamical network

In this paper, we investigate the locally and globally adaptive synchronization problem for an uncertain complex dynamical network with time-varying coupling delays based on the decentralized control. The coupling terms here are bounded by high-order polynomials with known gains that are ubiquitous in a large class of complex dynamical networks. We generalize the usual technology of searching for an appropriate coordinates transformation to change the network dynamics into a series of decoupled lower-dimensional systems. Several adaptive synchronization criteria are derived by constructing the Lyapunov-Krasovskii functional and Barbalat lemma, and the proposed criteria are simple in form and convenient for the practical engineering design. Numerical simulations illustrated by a nearest-neighbor coupling network verify the effectiveness of the proposed synchronization scheme.

Theory and model of water resources complex adaptive allocation system

Complex adaptive sys tem theory is a new and important embranchment of system science, which prov ides a new thought to research water resources allocation system. Based on the a nalysis of complexity and complex adaptive mechanism of water resources allocat ion system, a fire-new analysis model is presented in this paper. With t he description of dynamical mechanism of system, behavior characters of agents and the evaluation method of system status, an integrity research system is built to analyse the evolvement rule of water resources allocation system. A nd a brief research for the impact of water resources allocation in benefi cial regions of the Water Transfer from South to North China Project is conducted.

Dynamic modeling and aperiodically intermittent strategy for adaptive finite-time synchronization control of the multi-weighted complex transportation networks with multiple delays

The idea of network splitting according to time delay and weight is introduced.Based on the cyber physical systems(CPS),a class of multi-weighted complex transportation networks with multiple delays is modeled.The finite-time synchronization of the proposed complex transportation networks model is studied systematically.On the basis of the theory of stability,the technique of adaptive control,aperiodically intermittent control and finite-time control,the aperiodically intermittent adaptive finite-time synchronization controller is designed.The controller designed in this paper is beneficial for understanding the synchronization in multi-weighted complex transportation networks with multiple delays.In addition,the conditions for the existence of finite time synchronization have been discussed in detail.And the specific value of the settling finite time for synchronization is obtained.Moreover,the outer coupling configuration matrices are not required to be irreducible or symmetric.Finally,simulation results of the finite-time synchronization problem are given to illustrate the correctness of the results obtained.

Linguistic Reflection on the Online Catchword of Appreciation —From the Perspective of Usage-based Language Model and Complex Adaptive System Theory

Language is a special social phenomenon and is always on the changing process with the development of society. During the evolving process of language, new language varieties will continuously emerge due to the changes of some social and cultural factors. Cyber language is universally accepted as one type of the social language varieties. Basically, cyber language can be treated as a complex adaptive system which is influenced by the interaction between users’ cognition, social culture and the surrounding environments. Thus it is safe to say that cyber language is always undergoing a dynamic evolving process. With the usage-based language model as the theoretical foundation, this paper proposes a Complex Adaptive System (CAS) approach to analyze the expression of Appreciation to explore the complex, dynamic and nonlinear development of cyber language from the angle of meaning construction, grammaticalization and functional adaption respectively. It is found that the expression of Appreciation is experiencing adaptively a semantic connotations development and a process of grammatical functions expansion as well. This paper suggests that the emergence and development of cyber language is a novel and trendy social language phenomenon. Network language can achieve its process and evolution under the huge impact of social changes and social promotions. When faced with the changing surroundings, cyber language itself enjoys a timely adaption and responsive development to keep up with the new environments, which reflects the basic principle of language development, namely, language changes with the development of society.

CAS理论下的职业教育强适应性EMSV指标模型研究

增强职业教育适应性,仅有愿景描绘是不够的,亟需进一步标定“强”的靶向,用以指明“增强”路向,为此构建强适应性概念框架。创新之处在于突破“群氓”运动思维,从CAS理论视角重新审视强适应性生成逻辑,提出“强”的能量蕴含在多元适应性主体趋利而聚的复杂适应系统中,是利益共同体内部合作势能在时间轴上从隐性量变到显性质变并最终集中爆发的涌现结果这一全新观点,并创建EMSV指标模型这一强适应性概念框架:激励(E)为各方聚集提供支点,动力(M)是夯实利益共同体的基础,合力(S)是合作剩余的现实呈现,活力(V)是合力的双向持续表达。EMSV指标模型为“增强职业教育适应性”提供新理念、总原则、元规则、主标准的路向指引。

气候模式CAS-ESM-C对1月热带太平洋流场模态的模拟评估

利用气候模式CAS-ESM-C从1922年起84年的模拟资料,对1月份热带太平洋上层流场作复EOF分解及小波分析,并与实况以及理论解析解作对比讨论,以考察模式对赤道大洋上层流场的模拟能力,得到主要结论:(1)复EOF分解前3个模态的方差贡献为53.5%、12.9%、9.5%,累积方差贡献为75.9%,累积方差贡献比实况更高。(2)第一、二模态空间场与实况相比总体相像,流场都为赤道所俘获,在俘获区内的流场均以偏纬向流为主;差异在于模拟资料分析的赤道俘获区范围较实况要大,流场的经向流分量及越赤道流也较实况明显。(3)第一、二模态实时间系数序列无线性变化趋势,而实况则有。复EOF模态年际及年代际变化与实况相同或相近;第一、二模态中3~7年的年际变化是厄尔尼诺与南方涛动(ENSO)的反映;第一模态22~23年的年代际变化受北太平洋主要气候模态北太平洋年代际振荡(PDO)对热带太平洋的影响,而第二模态13年的年代际变化是受北太平洋次要气候模态北太平洋环流振荡(NPGO)对热带太平洋的影响;第一、二模态还都有峰值16年的年代际变化,这可能与印尼穿越流有关。(4)模拟资料分析的结果具有理论解析解中流场为赤道所俘获及流场为纬向流的特点,只是解析解中因无风应力强迫,流场呈纯纬向流。(5)第一(二)模态在赤道太平洋东部(中部)有海温动力异常,并可称之为东(中)部型ENSO模态。气候模式CAS-ESM-C对热带太平洋上层流场的模拟表现较佳。

基于CAS理论的都江堰景区疫后发展研究

为顺利实现都江堰水利风景区疫后恢复发展,结合复杂适应系统(CAS)理论,探究水利风景区政府、企业、管理者、消费者四大主体的适应机制,并运用刺激-反应模型,梳理都江堰水利风景区疫后发展出现的参与主体单一、业务开展受限、文娱服务功能减弱和经济增长动力不足等问题,提出主体主动适配客观环境、维护系统内部稳定性、统筹内外发展要素、创新平台展示等对策,为都江堰水利风景区的疫后发展提供参考。

CAS理论视角下远程分级诊疗服务运行机制研究

分级诊疗是我国5项基本医疗卫生制度之首,也是我国深化医药卫生体制改革的核心目标之一,有利于优化资源配置、提高医疗机构服务效率、促进卫生健康事业发展。利用复杂适应系统(Complex Adaptive System,CAS)理论,结合远程医疗技术,明确远程分级诊疗服务的需求方、供给方、行政监管方等参与主体,并利用其7个基本特征,从不同主体角度构建了远程分级诊疗服务的运行机制,从理论上为远程分级诊疗服务的发展提供新的思路。

Decentralized Adaptive Strategies for Synchronization of Fractional-order Complex Networks

This paper focuses on synchronization of fractionalorder complex dynamical networks with decentralized adaptive coupling.Based on local information among neighboring nodes,two fractional-order decentralized adaptive strategies are designed to tune all or only a small fraction of the coupling gains respectively.By constructing quadratic Lyapunov functions and utilizing fractional inequality techniques,Mittag-Leffler function,and Laplace transform,two sufficient conditions are derived for reaching network synchronization by using the proposed adaptive laws.Finally,two numerical examples are given to verify the theoretical results.

Understanding leukemic hematopoiesis as a complex adaptive system

Normal and abnormal hematopoiesis is working as a complex adaptive system. From this perspective, the development and the behavior of hematopoietic cell lineages appear as a balance between normal and abnormal hematopoiesis in the setting of a functioning or malfunctioning microenvironment under the control of the immune system and the influence of hereditary and environmental events.