Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

蛋白质ADP核糖基化对秦川牛肉品质的影响

为研究蛋白质ADP核糖基化对宰后成熟初期秦川牛肉线粒体功能及肉品质的影响,以20µmol/L Rucaparib(ADP核糖聚合酶1(poly(ADP-ribose)polymerase 1,PARP1)抑制剂)处理的秦川牛背最长肌为研究对象,测定贮藏0 h、6 h、12 h、2 d、4 d、8 d对照组和抑制剂处理组样品的线粒体相关指标及肌原纤维小片化指数(myofibril fragmentation index,MFI)、剪切力、pH值等品质指标,并采用免疫印迹(Western Blot)法检测PARP1、肌间线蛋白表达水平。结果表明:0 h~8 d(12 h除外)抑制剂组活性氧含量显著低于对照组(P<0.05),宰后0~12 h抑制剂组Caspase-3活性、MFI显著低于对照组(P<0.05);在2~4 d抑制剂组线粒体膜电位较对照组高,4~8 d抑制剂处理组琥珀酸脱氢酶(succinate dehydrogenase,SDH)活性显著高于对照组(P<0.05)。说明抑制表征ADP核糖基化反应进行的PARP1后,线粒体膜电位下降变缓,SDH活性升高,这一定程度上维持了线粒体功能,使MFI下降、肌间线蛋白降解变缓,从而延缓肉品嫩化。

基于有限时间ADP的微波加热高钛渣温度跟踪控制

针对常规控制方法对微波加热过程控制效果不够理想的问题,提出一种基于数据驱动模型的有限时间自适应动态规划微波加热温度跟踪算法。算法包含模型网络、评价网络和执行网络,这3个网络的实现依赖于神经网络。模型网络实现微波加热过程的数据驱动建模,评价网络和执行网络实现最优性能指标函数和控制功率的逼近。最后将温度跟踪转化为误差的镇定。通过理论推导证明了算法的收敛性及最优性,并进一步开展了微波加热高钛渣温度跟踪实验和仿真研究。结果表明,算法能有效地跟踪高钛渣的加热过程,基于ELMAN神经网络的模型预测误差小于1℃,温度跟踪误差小于0.2℃,在工业微波加热中具有潜在的应用价值。

Adaptive Optimal Discrete-Time Output-Feedback Using an Internal Model Principle and Adaptive Dynamic Programming

In order to address the output feedback issue for linear discrete-time systems, this work suggests a brand-new adaptive dynamic programming(ADP) technique based on the internal model principle(IMP). The proposed method, termed as IMP-ADP, does not require complete state feedback-merely the measurement of input and output data. More specifically, based on the IMP, the output control problem can first be converted into a stabilization problem. We then design an observer to reproduce the full state of the system by measuring the inputs and outputs. Moreover, this technique includes both a policy iteration algorithm and a value iteration algorithm to determine the optimal feedback gain without using a dynamic system model. It is important that with this concept one does not need to solve the regulator equation. Finally, this control method was tested on an inverter system of grid-connected LCLs to demonstrate that the proposed method provides the desired performance in terms of both tracking and disturbance rejection.

In vivo astrocyte reprogramming following spinal cord injury

Harmful and helpful roles of astrocytes in spinal cord injury(SCI):SCI induce gradable sensory,motor and autonomic impairments that correlate with the lesion severity and the rostro-caudal location of the injury site.The absence of spontaneous axonal regeneration after injury results from neuron-intrinsic and neuron-extrinsic parameters.Indeed,not only adult neurons display limited capability to regrow axons but also the injury environment contains inhibitors to axonal regeneration and a lack of growth-promoting factors.Amongst other cell populations that respond to the lesion,reactive astrocytes were first considered as only detrimental to spontaneous axonal regeneration.Indeed,astrocytes.

Transcriptional reprogramming during human osteoclast differentiation identifies regulators of osteoclast activity

Enhanced osteoclastogenesis and osteoclast activity contribute to the development of osteoporosis,which is characterized by increased bone resorption and inadequate bone formation.As novel antiosteoporotic therapeutics are needed,understanding the genetic regulation of human osteoclastogenesis could help identify potential treatment targets.This study aimed to provide an overview of transcriptional reprogramming during human osteoclast differentiation.Osteoclasts were differentiated from CD14+monocytes from eight female donors.RNA sequencing during differentiation revealed 8980 differentially expressed genes grouped into eight temporal patterns conserved across donors.These patterns revealed distinct molecular functions associated with postmenopausal osteoporosis susceptibility genes based on RNA from iliac crest biopsies and bone mineral density SNPs.Network analyses revealed mutual dependencies between temporal expression patterns and provided insight into subtype-specific transcriptional networks.The donor-specific expression patterns revealed genes at the monocyte stage,such as filamin B(FLNB)and oxidized low-density lipoprotein receptor 1(OLR1,encoding LOX-1),that are predictive of the resorptive activity of mature osteoclasts.The expression of differentially expressed G-protein coupled receptors was strong during osteoclast differentiation,and these receptors are associated with bone mineral density SNPs,suggesting that they play a pivotal role in osteoclast differentiation and activity.The regulatory effects of three differentially expressed G-protein coupled receptors were exemplified by in vitro pharmacological modulation of complement 5 A receptor 1(C5AR1),somatostatin receptor 2(SSTR2),and free fatty acid receptor 4(FFAR4/GPR120).Activating C5AR1 enhanced osteoclast formation,while activating SSTR2 decreased the resorptive activity of mature osteoclasts,and activating FFAR4 decreased both the number and resorptive activity of mature osteoclasts.In conclusion,we report the occurrence of transcriptional reprogramming during human osteoclast differentiation and identified SSTR2 and FFAR4 as antiresorptive G-protein coupled receptors and FLNB and LOX-1 as potential molecular markers of osteoclast activity.These data can help future investigations identify molecular regulators of osteoclast differentiation and activity and provide the basis for novel antiosteoporotic targets.

Uncertainty quantification of inverse analysis for geomaterials using probabilistic programming

Uncertainty is an essentially challenging for safe construction and long-term stability of geotechnical engineering.The inverse analysis is commonly utilized to determine the physico-mechanical parameters.However,conventional inverse analysis cannot deal with uncertainty in geotechnical and geological systems.In this study,a framework was developed to evaluate and quantify uncertainty in inverse analysis based on the reduced-order model(ROM)and probabilistic programming.The ROM was utilized to capture the mechanical and deformation properties of surrounding rock mass in geomechanical problems.Probabilistic programming was employed to evaluate uncertainty during construction in geotechnical engineering.A circular tunnel was then used to illustrate the proposed framework using analytical and numerical solution.The results show that the geomechanical parameters and associated uncertainty can be properly obtained and the proposed framework can capture the mechanical behaviors under uncertainty.Then,a slope case was employed to demonstrate the performance of the developed framework.The results prove that the proposed framework provides a scientific,feasible,and effective tool to characterize the properties and physical mechanism of geomaterials under uncertainty in geotechnical engineering problems.

Rapid Prototype Development Approach for Genetic Programming

Genetic Programming (GP) is an important approach to deal with complex problem analysis and modeling, and has been applied in a wide range of areas. The development of GP involves various aspects, including design of genetic operators, evolutionary controls and implementations of heuristic strategy, evaluations and other mechanisms. When designing genetic operators, it is necessary to consider the possible limitations of encoding methods of individuals. And when selecting evolutionary control strategies, it is also necessary to balance search efficiency and diversity based on representation characteristics as well as the problem itself. More importantly, all of these matters, among others, have to be implemented through tedious coding work. Therefore, GP development is both complex and time-consuming. To overcome some of these difficulties that hinder the enhancement of GP development efficiency, we explore the feasibility of mutual assistance among GP variants, and then propose a rapid GP prototyping development method based on πGrammatical Evolution (πGE). It is demonstrated through regression analysis experiments that not only is this method beneficial for the GP developers to get rid of some tedious implementations, but also enables them to concentrate on the essence of the referred problem, such as individual representation, decoding means and evaluation. Additionally, it provides new insights into the roles of individual delineations in phenotypes and semantic research of individuals.

三结构域蛋白8、Ras相关蛋白35、多聚ADP核糖聚合酶1在子宫内膜癌组织中的表达及检测临床意义

目的:探讨三结构域蛋白8(TRIM8)、Ras相关蛋白35(Rab35)、多聚ADP核糖聚合酶1(PARP-1)在子宫内膜癌(EC)组织中的表达及检测临床意义。方法:选择75例EC患者为研究对象,取手术切除的EC组织及癌旁正常子宫内膜组织,以免疫组织化学法检测EC组织和癌旁组织中TRIM8、Rab35、PARP-1蛋白表达情况,并结合患者临床病理因素分析其与临床病理特征的关系,随访5年,记录EC患者生存率,并分析EC患者预后影响因素。结果:Rab35、PARP-1蛋白在EC组织和癌旁组织中阳性表达率分别为71.00%和35.00%、74.00%和43.00%,TRIM8蛋白阳性表达率分别为30.00%和69.00%(均P<0.01)。EC组织中Rab35与PARP-1为正相关性,Rab35和PARP-1与TRIM8表达为负相关性(r=0.542、-0.461、-0.461,均P<0.05)。Rab35阳性患者5年生存率55.17%低于阴性患者70.57%,PARP-1阳性患者5年生存率65.96%低于阴性患者79.24%,TRIM8阳性患者5年生存率93.33%高于阴性患者72.73%(均P<0.05)。FIGO分期Ⅲ-Ⅳ期、肌层浸润深度≥1/2和PARP-1、TRIM8、Rab35表达异常为影响EC患者预后的危险因素(均P<0.05)。结论:子宫内膜癌组织PARP-1、Rab35表达升高、TRIM8表达降低,三者表达变化与子宫内膜癌的发生、发展有显著相关性。

A Dimensional Reduction Approach Based on Essential Constraints in Linear Programming

This paper presents a new dimension reduction strategy for medium and large-scale linear programming problems. The proposed method uses a subset of the original constraints and combines two algorithms: the weighted average and the cosine simplex algorithm. The first approach identifies binding constraints by using the weighted average of each constraint, whereas the second algorithm is based on the cosine similarity between the vector of the objective function and the constraints. These two approaches are complementary, and when used together, they locate the essential subset of initial constraints required for solving medium and large-scale linear programming problems. After reducing the dimension of the linear programming problem using the subset of the essential constraints, the solution method can be chosen from any suitable method for linear programming. The proposed approach was applied to a set of well-known benchmarks as well as more than 2000 random medium and large-scale linear programming problems. The results are promising, indicating that the new approach contributes to the reduction of both the size of the problems and the total number of iterations required. A tree-based classification model also confirmed the need for combining the two approaches. A detailed numerical example, the general numerical results, and the statistical analysis for the decision tree procedure are presented.

基于策略迭代ADP的碳纤维角联织机张力控制

针对碳纤维角联织机经纱张力控制问题,考虑开口等不确定因素对经纱张力的影响,建立了离散非线性送经系统张力控制模型,提出了策略迭代自适应动态规划(ADP),并对ADP中评价网络设计了自适应权值更新率;证明了策略迭代ADP在离散系统的收敛性,削减了非线性及不确定因素对经纱张力的影响,实现了对经纱张力的稳定控制,提高了系统鲁棒性。仿真结果表明:相比传统ADP,策略迭代ADP可以使经纱张力在2 s内快速无波动的到达稳定状态,使系统性能指标函数收敛更优。

聚ADP-核糖聚合酶-1在喉鳞状细胞癌细胞DNA氧化损伤及凋亡中的作用

目的探究聚ADP-核糖聚合酶-1(PARP1)在喉鳞状细胞癌(LSCC)中的表达及其对LSCC细胞DNA氧化损伤与凋亡的影响。方法收集40例患者的LSCC组织及癌旁组织样本,采用免疫组化染色和qRT-PCR检测LSCC组织与癌旁组织中PARP1的表达。采用不同浓度Menadione诱导LSCC细胞系Hep-2,MTT法检测不同浓度诱导下细胞增殖抑制率。将Hep-2细胞随机分为对照组(正常培养)、Men组(20μmol/L Menadione处理)、Men+PARP1抑制剂组(20μmol/L Menadione+10μmol/L PARP1抑制剂Olaparib共处理)。MTT法计算各组细胞增殖抑制率;流式细胞术检测各组细胞内活性氧(ROS)水平;细胞免疫荧光染色观察各组细胞DNA损伤标记分子γ-H2AX焦点生成情况;Western blot测定各组细胞γ-H2AX、DNA-PKcs、BRCA1、LIG4、Caspase-3及Caspase-9蛋白表达;Hoechst33258染色观察各组细胞凋亡情况。结果LSCC组织中PARP1阳性表达率及PARP1 mRNA表达均明显高于癌旁组织(P<0.01)。Hep-2细胞经不同浓度Menadione诱导后,细胞增殖抑制率均升高(P<0.05)。与对照组比较,Men组和Men+PARP1抑制剂组的细胞增殖抑制率升高,细胞ROS水平升高,γ-H2AX焦点形成明显增加,γ-H2AX蛋白表达上调,DNA-PKcs、BRCA1、LIG4蛋白表达均下调,细胞出现浓缩、碎裂的蓝色凋亡体,Caspase-3和Caspase-9蛋白表达上调,差异均有统计学意义(P<0.05),且Men+PARP1抑制剂组以上变化较Men组更明显(P<0.05)。结论PARP1在LSCC组织中高表达,抑制其表达能够进一步加重Menadione诱导的Hep-2细胞DNA氧化损伤,并促进细胞凋亡。

第一性原理研究Fe及其团簇缺陷对KDP和ADP晶体激光损伤的影响

由于金属杂质离子对晶体损伤性质有不容忽视的影响,受实验条件限制,Fe及其团簇缺陷对晶体的影响机制尚不明确。采用第一性原理的方法,对磷酸二氢钾(KDP)和磷酸二氢铵(ADP)晶体中的Fe及其团簇缺陷进行模拟研究,确定其对晶体结构及光学性质方面的影响。研究发现,Fe进入KDP和ADP晶体中主要以取代P原子形成FeO_(4)基团最稳定,且其稳定形式以Fe^(3+)为主。磁性状态研究发现磁性条件对晶体的结构和能量影响不大,Fe对晶体的损伤主要通过引起200~300 nm范围明显的光学吸收影响损伤阈值。Fe进入晶体中形成团簇缺陷可通过电荷补偿与O空位(VO)复合,几乎不会与OH空位(VOH)复合,团簇缺陷以Fe对晶体结构和性质的影响为主。

基于ADP的一类未知非线性系统事件触发输出反馈最优控制

针对一类未知非线性系统,提出基于ADP(Adaptive Dynamic Programming)的事件触发输出反馈最优控制策略,此方法只用到了系统的输出量。考虑到系统内部状态量无法测量和系统模型难以获得的问题,设计神经网络状态观测器来估计系统的不可测状态量并通过输出信息重构了系统的内部状态。在获得系统的未知动态和状态量信息后,设计结合事件触发技术的ADP输出反馈最优控制策略。通过Lyapunov理论推导了神经网络观测器和评估网络的权值更新率,并且证明了闭环控制系统的稳定性。通过仿真实验验证了该控制方法的有效性。

靶向聚ADP核糖聚合酶抑制剂AZD-9574的合成工艺改进

目的精简合成工艺,优化原始步骤,对靶向聚ADP核糖聚合酶抑制剂AZD-9574合成工艺进行改进。方法本研究分别以5-溴代吡啶酸甲酯和2-甲基(4-溴-3-氟-2-硝基苯胺)丙酸酯为起始原料,经氟代、取代、氨解、脱保护基、取代、还原环化、氧化、偶联、取代合成目标物靶向聚ADP核糖聚合酶抑制剂。结果对合成中关键步骤反应条件进行了优化,提高了产率。结论该方法操作简便、原料价廉易得、反应条件温和,可用于靶向聚ADP核糖聚合酶抑制剂的大规模制备。

Event-based performance guaranteed tracking control for constrained nonlinear system via adaptive dynamic programming method

An optimal tracking control problem for a class of nonlinear systems with guaranteed performance and asymmetric input constraints is discussed in this paper.The control policy is implemented by adaptive dynamic programming(ADP)algorithm under two event-based triggering mechanisms.It is often challenging to design an optimal control law due to the system deviation caused by asymmetric input constraints.First,a prescribed performance control technique is employed to guarantee the tracking errors within predetermined boundaries.Subsequently,considering the asymmetric input constraints,a discounted non-quadratic cost function is introduced.Moreover,in order to reduce controller updates,an event-triggered control law is developed for ADP algorithm.After that,to further simplify the complexity of controller design,this work is extended to a self-triggered case for relaxing the need for continuous signal monitoring by hardware devices.By employing the Lyapunov method,the uniform ultimate boundedness of all signals is proved to be guaranteed.Finally,a simulation example on a mass–spring–damper system subject to asymmetric input constraints is provided to validate the effectiveness of the proposed control scheme.

Sequential quadratic programming-based non-cooperative target distributed hybrid processing optimization method

The distributed hybrid processing optimization problem of non-cooperative targets is an important research direction for future networked air-defense and anti-missile firepower systems. In this paper, the air-defense anti-missile targets defense problem is abstracted as a nonconvex constrained combinatorial optimization problem with the optimization objective of maximizing the degree of contribution of the processing scheme to non-cooperative targets, and the constraints mainly consider geographical conditions and anti-missile equipment resources. The grid discretization concept is used to partition the defense area into network nodes, and the overall defense strategy scheme is described as a nonlinear programming problem to solve the minimum defense cost within the maximum defense capability of the defense system network. In the solution of the minimum defense cost problem, the processing scheme, equipment coverage capability, constraints and node cost requirements are characterized, then a nonlinear mathematical model of the non-cooperative target distributed hybrid processing optimization problem is established, and a local optimal solution based on the sequential quadratic programming algorithm is constructed, and the optimal firepower processing scheme is given by using the sequential quadratic programming method containing non-convex quadratic equations and inequality constraints. Finally, the effectiveness of the proposed method is verified by simulation examples.

血清ADP、MA与非大血管闭塞急性缺血性脑卒中患者预后的相关性

目的 探讨血清二磷酸腺苷(ADP)、血栓弹力图中的最大振幅(MA)与非大血管闭塞急性缺血性脑卒中患者预后的相关性。方法 选取2019年1月至2022年5月在太和县人民医院神经内科进行就诊的缺血性脑卒中患者100例作为研究对象。100例患者以随机数字法分为观察组和对照组各50例,对照组50例接受阿司匹林治疗,观察组50例接受替罗非班治疗。依据两组预后情况(mRS评分)分为预后良好组和预后不良组,分析血清ADP、MA与非大血管闭塞急性缺血性脑卒中患者预后的关系。结果治疗3个月后,观察组血清ADP、MA均高于对照组,差异有统计学意义(t=2.830、7.844,P<0.05);预后良好组血清ADP、MA高于预后不良组,差异有统计学意义(t=3.313、2.287,P<0.05),MRS评分低于预后不良组,差异有统计学意义(t=19.685,P<0.05);Pearson相关性分析显示,血清ADP与mRS评分呈负相关(r=-0.272,P=0.006),与MA呈负相关(r=-0.203,P=0.043);Logistic回归分析显示,血清ADP、MA降低,MRS评分升高是非大血管闭塞急性缺血性脑卒中预后不良的独立影响因素(P<0.05);ROC曲线图显示,血清ADP的预测价值最高,AUC为0.725,敏感度为73.9%,特异度为64.5%。结论 非大血管闭塞急性缺血性脑卒中应用替罗非班治疗的效果显著,血清ADP、MA是非大血管闭塞急性缺血性脑卒中患者预后不良的独立微信因素,其中MA对非大血管闭塞急性缺血性脑卒中预后不良具有较高预测价值。

Generating oligodendroglia from adult mesenchymal cells for transplantation:cell reprogramming or direct lineage conversion?

Cell replacement therapy has long been proposed as a treatment for the damaged nervous system.One of the most challenging aspects of such a strategy,however,is finding sources of donor cells for transplantation.Autologous neural cells are rarely an option as every cell in the nervous system has a defined function that would be lost if that cell was to be removed.One possibility would be sourcing precursor or differentiated cells from fetal tissues;however,aside from ethical issues,heterologous cells are at risk of immunological rejection in the long term.Methodological improvements over the past 15 years have led to the possibility that autologous non-neural cells could be used for cell transplantation through their conversion into neural derivatives.

Developmental exposure to thyroid disruptors:misprogramming of the brain's stem cells in later life?

Introduction:Ever since the discovery of neural stem cells(NSCs)in the adult mammalian brain,scientists have been trying to decipher which signals govern their turnover and lineage commitment to generate neurons and glia.Understanding their role in nervous tissue homeostasis can provide new insights into the etiology of several neurological disorders,and might one day be turned to our advantage to promote endogenous brain injury repair.Others and we have identified thyroid hormone(TH)as a key factor transcriptionally regulating NSC behavior in the largest niche of the adult mammalian brain:the subventricular zone(SVZ).