Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

In vivo astrocyte reprogramming following spinal cord injury

Harmful and helpful roles of astrocytes in spinal cord injury(SCI):SCI induce gradable sensory,motor and autonomic impairments that correlate with the lesion severity and the rostro-caudal location of the injury site.The absence of spontaneous axonal regeneration after injury results from neuron-intrinsic and neuron-extrinsic parameters.Indeed,not only adult neurons display limited capability to regrow axons but also the injury environment contains inhibitors to axonal regeneration and a lack of growth-promoting factors.Amongst other cell populations that respond to the lesion,reactive astrocytes were first considered as only detrimental to spontaneous axonal regeneration.Indeed,astrocytes.

Transcriptional reprogramming during human osteoclast differentiation identifies regulators of osteoclast activity

Enhanced osteoclastogenesis and osteoclast activity contribute to the development of osteoporosis,which is characterized by increased bone resorption and inadequate bone formation.As novel antiosteoporotic therapeutics are needed,understanding the genetic regulation of human osteoclastogenesis could help identify potential treatment targets.This study aimed to provide an overview of transcriptional reprogramming during human osteoclast differentiation.Osteoclasts were differentiated from CD14+monocytes from eight female donors.RNA sequencing during differentiation revealed 8980 differentially expressed genes grouped into eight temporal patterns conserved across donors.These patterns revealed distinct molecular functions associated with postmenopausal osteoporosis susceptibility genes based on RNA from iliac crest biopsies and bone mineral density SNPs.Network analyses revealed mutual dependencies between temporal expression patterns and provided insight into subtype-specific transcriptional networks.The donor-specific expression patterns revealed genes at the monocyte stage,such as filamin B(FLNB)and oxidized low-density lipoprotein receptor 1(OLR1,encoding LOX-1),that are predictive of the resorptive activity of mature osteoclasts.The expression of differentially expressed G-protein coupled receptors was strong during osteoclast differentiation,and these receptors are associated with bone mineral density SNPs,suggesting that they play a pivotal role in osteoclast differentiation and activity.The regulatory effects of three differentially expressed G-protein coupled receptors were exemplified by in vitro pharmacological modulation of complement 5 A receptor 1(C5AR1),somatostatin receptor 2(SSTR2),and free fatty acid receptor 4(FFAR4/GPR120).Activating C5AR1 enhanced osteoclast formation,while activating SSTR2 decreased the resorptive activity of mature osteoclasts,and activating FFAR4 decreased both the number and resorptive activity of mature osteoclasts.In conclusion,we report the occurrence of transcriptional reprogramming during human osteoclast differentiation and identified SSTR2 and FFAR4 as antiresorptive G-protein coupled receptors and FLNB and LOX-1 as potential molecular markers of osteoclast activity.These data can help future investigations identify molecular regulators of osteoclast differentiation and activity and provide the basis for novel antiosteoporotic targets.

Improved Unit Commitment with Accurate Dynamic Scenarios Clustering Based on Multi-Parametric Programming and Benders Decomposition

Stochastic unit commitment is one of the most powerful methods to address uncertainty. However, the existingscenario clustering technique for stochastic unit commitment cannot accurately select representative scenarios,which threatens the robustness of stochastic unit commitment and hinders its application. This paper providesa stochastic unit commitment with dynamic scenario clustering based on multi-parametric programming andBenders decomposition. The stochastic unit commitment is solved via the Benders decomposition, which decouplesthe primal problem into the master problem and two types of subproblems. In the master problem, the committedgenerator is determined, while the feasibility and optimality of generator output are checked in these twosubproblems. Scenarios are dynamically clustered during the subproblem solution process through the multiparametric programming with respect to the solution of the master problem. In other words, multiple scenariosare clustered into several representative scenarios after the subproblem is solved, and the Benders cut obtainedby the representative scenario is generated for the master problem. Different from the conventional stochasticunit commitment, the proposed approach integrates scenario clustering into the Benders decomposition solutionprocess. Such a clustering approach could accurately cluster representative scenarios that have impacts on theunit commitment. The proposed method is tested on a 6-bus system and the modified IEEE 118-bus system.Numerical results illustrate the effectiveness of the proposed method in clustering scenarios. Compared withthe conventional clustering method, the proposed method can accurately select representative scenarios whilemitigating computational burden, thus guaranteeing the robustness of unit commitment.

Uncertainty quantification of inverse analysis for geomaterials using probabilistic programming

Uncertainty is an essentially challenging for safe construction and long-term stability of geotechnical engineering.The inverse analysis is commonly utilized to determine the physico-mechanical parameters.However,conventional inverse analysis cannot deal with uncertainty in geotechnical and geological systems.In this study,a framework was developed to evaluate and quantify uncertainty in inverse analysis based on the reduced-order model(ROM)and probabilistic programming.The ROM was utilized to capture the mechanical and deformation properties of surrounding rock mass in geomechanical problems.Probabilistic programming was employed to evaluate uncertainty during construction in geotechnical engineering.A circular tunnel was then used to illustrate the proposed framework using analytical and numerical solution.The results show that the geomechanical parameters and associated uncertainty can be properly obtained and the proposed framework can capture the mechanical behaviors under uncertainty.Then,a slope case was employed to demonstrate the performance of the developed framework.The results prove that the proposed framework provides a scientific,feasible,and effective tool to characterize the properties and physical mechanism of geomaterials under uncertainty in geotechnical engineering problems.

Crosstalk among mitophagy,pyroptosis,ferroptosis,and necroptosis in central nervous system injuries

Central nervous system injuries have a high rate of resulting in disability and mortality;however,at present,effective treatments are lacking.Programmed cell death,which is a genetically determined fo rm of active and ordered cell death with many types,has recently attra cted increasing attention due to its functions in determining the fate of cell survival.A growing number of studies have suggested that programmed cell death is involved in central nervous system injuries and plays an important role in the progression of brain damage.In this review,we provide an ove rview of the role of programmed cell death in central nervous system injuries,including the pathways involved in mitophagy,pyroptosis,ferroptosis,and necroptosis,and the underlying mechanisms by which mitophagy regulates pyroptosis,ferroptosis,and necro ptosis.We also discuss the new direction of therapeutic strategies to rgeting mitophagy for the treatment of central nervous system injuries,with the aim to determine the connection between programmed cell death and central nervous system injuries and to identify new therapies to modulate programmed cell death following central nervous system injury.In conclusion,based on these properties and effects,interventions targeting programmed cell death could be developed as potential therapeutic agents for central nervous system injury patients.

Optimal Bidding Strategies of Microgrid with Demand Side Management for Economic Emission Dispatch Incorporating Uncertainty and Outage of Renewable Energy Sources

In the restructured electricity market,microgrid(MG),with the incorporation of smart grid technologies,distributed energy resources(DERs),a pumped-storage-hydraulic(PSH)unit,and a demand response program(DRP),is a smarter and more reliable electricity provider.DER consists of gas turbines and renewable energy sources such as photovoltaic systems and wind turbines.Better bidding strategies,prepared by MG operators,decrease the electricity cost and emissions from upstream grid and conventional and renewable energy sources(RES).But it is inefficient due to the very high sporadic characteristics of RES and the very high outage rate.To solve these issues,this study suggests non-dominated sorting genetic algorithm Ⅱ(NSGA-Ⅱ)for an optimal bidding strategy considering pumped hydroelectric energy storage and DRP based on outage conditions and uncertainties of renewable energy sources.The uncertainty related to solar and wind units is modeled using lognormal and Weibull probability distributions.TOU-based DRP is used,especially considering the time of outages along with the time of peak loads and prices,to enhance the reliability of MG and reduce costs and emissions.

Transient Stability Preventive Control of Wind Farm Connected Power System Considering the Uncertainty

To address uncertainty as well as transient stability constraints simultaneously in the preventive control of windfarm systems, a novel three-stage optimization strategy is established in this paper. In the first stage, the probabilisticmulti-objective particle swarm optimization based on the point estimate method is employed to cope with thestochastic factors. The transient security region of the system is accurately ensured by the interior point methodin the second stage. Finally, the verification of the final optimal objectives and satisfied constraints are enforcedin the last stage. Furthermore, the proposed strategy is a general framework that can combine other optimizationalgorithms. The proposed methodology is tested on the modified WSCC 9-bus system and the New England 39-bussystem. The results verify the feasibility of the method.

Rapid Prototype Development Approach for Genetic Programming

Genetic Programming (GP) is an important approach to deal with complex problem analysis and modeling, and has been applied in a wide range of areas. The development of GP involves various aspects, including design of genetic operators, evolutionary controls and implementations of heuristic strategy, evaluations and other mechanisms. When designing genetic operators, it is necessary to consider the possible limitations of encoding methods of individuals. And when selecting evolutionary control strategies, it is also necessary to balance search efficiency and diversity based on representation characteristics as well as the problem itself. More importantly, all of these matters, among others, have to be implemented through tedious coding work. Therefore, GP development is both complex and time-consuming. To overcome some of these difficulties that hinder the enhancement of GP development efficiency, we explore the feasibility of mutual assistance among GP variants, and then propose a rapid GP prototyping development method based on πGrammatical Evolution (πGE). It is demonstrated through regression analysis experiments that not only is this method beneficial for the GP developers to get rid of some tedious implementations, but also enables them to concentrate on the essence of the referred problem, such as individual representation, decoding means and evaluation. Additionally, it provides new insights into the roles of individual delineations in phenotypes and semantic research of individuals.

Short-term displacement prediction for newly established monitoring slopes based on transfer learning

This study makes a significant progress in addressing the challenges of short-term slope displacement prediction in the Universal Landslide Monitoring Program,an unprecedented disaster mitigation program in China,where lots of newly established monitoring slopes lack sufficient historical deformation data,making it difficult to extract deformation patterns and provide effective predictions which plays a crucial role in the early warning and forecasting of landslide hazards.A slope displacement prediction method based on transfer learning is therefore proposed.Initially,the method transfers the deformation patterns learned from slopes with relatively rich deformation data by a pre-trained model based on a multi-slope integrated dataset to newly established monitoring slopes with limited or even no useful data,thus enabling rapid and efficient predictions for these slopes.Subsequently,as time goes on and monitoring data accumulates,fine-tuning of the pre-trained model for individual slopes can further improve prediction accuracy,enabling continuous optimization of prediction results.A case study indicates that,after being trained on a multi-slope integrated dataset,the TCN-Transformer model can efficiently serve as a pretrained model for displacement prediction at newly established monitoring slopes.The three-day average RMSE is significantly reduced by 34.6%compared to models trained only on individual slope data,and it also successfully predicts the majority of deformation peaks.The fine-tuned model based on accumulated data on the target newly established monitoring slope further reduced the three-day RMSE by 37.2%,demonstrating a considerable predictive accuracy.In conclusion,taking advantage of transfer learning,the proposed slope displacement prediction method effectively utilizes the available data,which enables the rapid deployment and continual refinement of displacement predictions on newly established monitoring slopes.

Adaptive Optimal Discrete-Time Output-Feedback Using an Internal Model Principle and Adaptive Dynamic Programming

In order to address the output feedback issue for linear discrete-time systems, this work suggests a brand-new adaptive dynamic programming(ADP) technique based on the internal model principle(IMP). The proposed method, termed as IMP-ADP, does not require complete state feedback-merely the measurement of input and output data. More specifically, based on the IMP, the output control problem can first be converted into a stabilization problem. We then design an observer to reproduce the full state of the system by measuring the inputs and outputs. Moreover, this technique includes both a policy iteration algorithm and a value iteration algorithm to determine the optimal feedback gain without using a dynamic system model. It is important that with this concept one does not need to solve the regulator equation. Finally, this control method was tested on an inverter system of grid-connected LCLs to demonstrate that the proposed method provides the desired performance in terms of both tracking and disturbance rejection.

A Dimensional Reduction Approach Based on Essential Constraints in Linear Programming

This paper presents a new dimension reduction strategy for medium and large-scale linear programming problems. The proposed method uses a subset of the original constraints and combines two algorithms: the weighted average and the cosine simplex algorithm. The first approach identifies binding constraints by using the weighted average of each constraint, whereas the second algorithm is based on the cosine similarity between the vector of the objective function and the constraints. These two approaches are complementary, and when used together, they locate the essential subset of initial constraints required for solving medium and large-scale linear programming problems. After reducing the dimension of the linear programming problem using the subset of the essential constraints, the solution method can be chosen from any suitable method for linear programming. The proposed approach was applied to a set of well-known benchmarks as well as more than 2000 random medium and large-scale linear programming problems. The results are promising, indicating that the new approach contributes to the reduction of both the size of the problems and the total number of iterations required. A tree-based classification model also confirmed the need for combining the two approaches. A detailed numerical example, the general numerical results, and the statistical analysis for the decision tree procedure are presented.

YKK's New DynaPel^(TM) Water-Repellant Zipper Wins Best Product in ISPO

The new DynaPel^(TM) zipper uses GTT EMPEL® technology to eliminate the need for a PU film Atlanta,GA (Dec 13,2023)-The ISPO Textrends judges have selected YKK’s DynaPel^(TM) water-repellent zipper as the Best Product in the accessories category.The competition,held twice a year in conjunction with the ISPO trade show,recognizes the most innovative performance textiles,components,and apparel.

ISPO Beijing 2024户外产业呈蓬勃势头 新品齐聚展品牌多元风貌

ISPO Beijing 2024亚洲运动用品与时尚展于1月12-14日在北京国家会议中心举办,共吸引了来自近500家参展商的700余个国内外品牌参展,迎来超3万名行业观众和运动爱好者。在开幕式上,慕尼黑展览上海有限公司总经理徐佳表示:“经历了2023年的重启与新生,很高兴再次看到许多老朋友来展示过去一年的迭代成果,也更欣慰有诸多新面孔来到这里交流所得、达成合作。加速体育产业变革是ISPO进入中国20年来始终秉持的理念,ISPO将进一步促进体育和文化、旅游、商业的融合,关注产业趋势变化,持续为业态更新、品牌与产品的迭代及从业者与爱好者的交流提供平台。”

Developmental exposure to thyroid disruptors:misprogramming of the brain's stem cells in later life?

Introduction:Ever since the discovery of neural stem cells(NSCs)in the adult mammalian brain,scientists have been trying to decipher which signals govern their turnover and lineage commitment to generate neurons and glia.Understanding their role in nervous tissue homeostasis can provide new insights into the etiology of several neurological disorders,and might one day be turned to our advantage to promote endogenous brain injury repair.Others and we have identified thyroid hormone(TH)as a key factor transcriptionally regulating NSC behavior in the largest niche of the adult mammalian brain:the subventricular zone(SVZ).

Event-based performance guaranteed tracking control for constrained nonlinear system via adaptive dynamic programming method

An optimal tracking control problem for a class of nonlinear systems with guaranteed performance and asymmetric input constraints is discussed in this paper.The control policy is implemented by adaptive dynamic programming(ADP)algorithm under two event-based triggering mechanisms.It is often challenging to design an optimal control law due to the system deviation caused by asymmetric input constraints.First,a prescribed performance control technique is employed to guarantee the tracking errors within predetermined boundaries.Subsequently,considering the asymmetric input constraints,a discounted non-quadratic cost function is introduced.Moreover,in order to reduce controller updates,an event-triggered control law is developed for ADP algorithm.After that,to further simplify the complexity of controller design,this work is extended to a self-triggered case for relaxing the need for continuous signal monitoring by hardware devices.By employing the Lyapunov method,the uniform ultimate boundedness of all signals is proved to be guaranteed.Finally,a simulation example on a mass–spring–damper system subject to asymmetric input constraints is provided to validate the effectiveness of the proposed control scheme.

面向深度学习视觉应用ISP过程的图像缩放攻击实验平台

大多数基于深度学习技术的视觉设备都配备了图像信号处理(Image Signal Processing, ISP)过程以实现RAW数据到RGB图像的转换,同时集成数据预处理过程以完成高效的图像处理。该文在同时考虑ISP和数据预处理过程影响基础上,搭建了深度学习视觉应用实验平台,提出针对ISP过程的图像缩放攻击,即精心制作对抗RAW经过ISP过程得到攻击图像,一旦缩放到特定尺寸就会呈现出完全不同的样貌。由于所提出的攻击是由基于ISP过程的梯度驱动的,因此构建了一个等效模型来学习目标ISP的转换过程,利用等效模型的近似梯度来发动攻击。该攻击平台的构建涵盖了深度学习算法、图像处理及对抗攻击优化等内容,有助于学生深入学习和理解基于深度学习视觉应用的任务处理原理以及深度学习模型的弱点,培养学生针对复杂算法问题的创新与实践能力。

关于ISP锌冶炼工艺转型升级和绿色发展的几点思考

本文分享了近年来我国ISP锌冶炼工艺取得的进展以及转型升级和绿色发展的几点思考。目前,我国铅锌冶炼技术比较成熟,铅冶炼以侧吹或底吹熔池熔炼为主的三连炉工艺,锌冶炼以湿法浸出为主的工艺,在环保和能耗方面都处于国际领先水平。密闭鼓风炉炼铅锌工艺(以下简称“ISP工艺”)则是一种比较成熟的铅锌联合冶炼方法,也是当今世界最主要的火法炼锌工艺,具有原料适应性强、综合回收利用水平高、经济效益好、锌锭品质高等优势,但同时也因能耗较高、环保风险大等问题受到广泛关注。

Policy Iteration for Optimal Control of Discrete-Time Time-Varying Nonlinear Systems

Aimed at infinite horizon optimal control problems of discrete time-varying nonlinear systems,in this paper,a new iterative adaptive dynamic programming algorithm,which is the discrete-time time-varying policy iteration(DTTV)algorithm,is developed.The iterative control law is designed to update the iterative value function which approximates the index function of optimal performance.The admissibility of the iterative control law is analyzed.The results show that the iterative value function is non-increasingly convergent to the Bellman-equation optimal solution.To implement the algorithm,neural networks are employed and a new implementation structure is established,which avoids solving the generalized Bellman equation in each iteration.Finally,the optimal control laws for torsional pendulum and inverted pendulum systems are obtained by using the DTTV policy iteration algorithm,where the mass and pendulum bar length are permitted to be time-varying parameters.The effectiveness of the developed method is illustrated by numerical results and comparisons.

Coordinated planning for flexible interconnection and energy storage system in low-voltage distribution networks to improve the accommodation capacity of photovoltaic

The increasing proportion of distributed photovoltaics(DPVs)and electric vehicle charging stations in low-voltage distribution networks(LVDNs)has resulted in challenges such as distribution transformer overloads and voltage violations.To address these problems,we propose a coordinated planning method for flexible interconnections and energy storage systems(ESSs)to improve the accommodation capacity of DPVs.First,the power-transfer characteristics of flexible interconnection and ESSs are analyzed.The equipment costs of the voltage source converters(VSCs)and ESSs are also analyzed comprehensively,considering the differences in installation and maintenance costs for different installation locations.Second,a bilevel programming model is established to minimize the annual comprehensive cost and yearly total PV curtailment capacity.Within this framework,the upper-level model optimizes the installation locations and capacities of the VSCs and ESSs,whereas the lower-level model optimizes the operating power of the VSCs and ESSs.The proposed model is solved using a non-dominated sorting genetic algorithm with an elite strategy(NSGA-II).The effectiveness of the proposed planning method is validated through an actual LVDN scenario,which demonstrates its advantages in enhancing PV accommodation capacity.In addition,the economic benefits of various planning schemes with different flexible interconnection topologies and different PV grid-connected forms are quantitatively analyzed,demonstrating the adaptability of the proposed coordinated planning method.