Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Crosstalk among mitophagy,pyroptosis,ferroptosis,and necroptosis in central nervous system injuries

Central nervous system injuries have a high rate of resulting in disability and mortality;however,at present,effective treatments are lacking.Programmed cell death,which is a genetically determined fo rm of active and ordered cell death with many types,has recently attra cted increasing attention due to its functions in determining the fate of cell survival.A growing number of studies have suggested that programmed cell death is involved in central nervous system injuries and plays an important role in the progression of brain damage.In this review,we provide an ove rview of the role of programmed cell death in central nervous system injuries,including the pathways involved in mitophagy,pyroptosis,ferroptosis,and necroptosis,and the underlying mechanisms by which mitophagy regulates pyroptosis,ferroptosis,and necro ptosis.We also discuss the new direction of therapeutic strategies to rgeting mitophagy for the treatment of central nervous system injuries,with the aim to determine the connection between programmed cell death and central nervous system injuries and to identify new therapies to modulate programmed cell death following central nervous system injury.In conclusion,based on these properties and effects,interventions targeting programmed cell death could be developed as potential therapeutic agents for central nervous system injury patients.

Uncertainty quantification of inverse analysis for geomaterials using probabilistic programming

Uncertainty is an essentially challenging for safe construction and long-term stability of geotechnical engineering.The inverse analysis is commonly utilized to determine the physico-mechanical parameters.However,conventional inverse analysis cannot deal with uncertainty in geotechnical and geological systems.In this study,a framework was developed to evaluate and quantify uncertainty in inverse analysis based on the reduced-order model(ROM)and probabilistic programming.The ROM was utilized to capture the mechanical and deformation properties of surrounding rock mass in geomechanical problems.Probabilistic programming was employed to evaluate uncertainty during construction in geotechnical engineering.A circular tunnel was then used to illustrate the proposed framework using analytical and numerical solution.The results show that the geomechanical parameters and associated uncertainty can be properly obtained and the proposed framework can capture the mechanical behaviors under uncertainty.Then,a slope case was employed to demonstrate the performance of the developed framework.The results prove that the proposed framework provides a scientific,feasible,and effective tool to characterize the properties and physical mechanism of geomaterials under uncertainty in geotechnical engineering problems.

Transcriptional reprogramming during human osteoclast differentiation identifies regulators of osteoclast activity

Enhanced osteoclastogenesis and osteoclast activity contribute to the development of osteoporosis,which is characterized by increased bone resorption and inadequate bone formation.As novel antiosteoporotic therapeutics are needed,understanding the genetic regulation of human osteoclastogenesis could help identify potential treatment targets.This study aimed to provide an overview of transcriptional reprogramming during human osteoclast differentiation.Osteoclasts were differentiated from CD14+monocytes from eight female donors.RNA sequencing during differentiation revealed 8980 differentially expressed genes grouped into eight temporal patterns conserved across donors.These patterns revealed distinct molecular functions associated with postmenopausal osteoporosis susceptibility genes based on RNA from iliac crest biopsies and bone mineral density SNPs.Network analyses revealed mutual dependencies between temporal expression patterns and provided insight into subtype-specific transcriptional networks.The donor-specific expression patterns revealed genes at the monocyte stage,such as filamin B(FLNB)and oxidized low-density lipoprotein receptor 1(OLR1,encoding LOX-1),that are predictive of the resorptive activity of mature osteoclasts.The expression of differentially expressed G-protein coupled receptors was strong during osteoclast differentiation,and these receptors are associated with bone mineral density SNPs,suggesting that they play a pivotal role in osteoclast differentiation and activity.The regulatory effects of three differentially expressed G-protein coupled receptors were exemplified by in vitro pharmacological modulation of complement 5 A receptor 1(C5AR1),somatostatin receptor 2(SSTR2),and free fatty acid receptor 4(FFAR4/GPR120).Activating C5AR1 enhanced osteoclast formation,while activating SSTR2 decreased the resorptive activity of mature osteoclasts,and activating FFAR4 decreased both the number and resorptive activity of mature osteoclasts.In conclusion,we report the occurrence of transcriptional reprogramming during human osteoclast differentiation and identified SSTR2 and FFAR4 as antiresorptive G-protein coupled receptors and FLNB and LOX-1 as potential molecular markers of osteoclast activity.These data can help future investigations identify molecular regulators of osteoclast differentiation and activity and provide the basis for novel antiosteoporotic targets.

Glia-to-neuron reprogramming to the rescue?

Over the last two decades,the dogma that cell fate is immutable has been increasingly challenged,with important implications for regenerative medicine.The brea kth rough discovery that induced pluripotent stem cells could be generated from adult mouse fibroblasts is powerful proof that cell fate can be changed.An exciting extension of the discovery of cell fate impermanence is the direct cellular reprogram ming hypothesis-that terminally differentiated cells can be reprogrammed into other adult cell fates without first passing through a stem cell state.

Improved Unit Commitment with Accurate Dynamic Scenarios Clustering Based on Multi-Parametric Programming and Benders Decomposition

Stochastic unit commitment is one of the most powerful methods to address uncertainty. However, the existingscenario clustering technique for stochastic unit commitment cannot accurately select representative scenarios,which threatens the robustness of stochastic unit commitment and hinders its application. This paper providesa stochastic unit commitment with dynamic scenario clustering based on multi-parametric programming andBenders decomposition. The stochastic unit commitment is solved via the Benders decomposition, which decouplesthe primal problem into the master problem and two types of subproblems. In the master problem, the committedgenerator is determined, while the feasibility and optimality of generator output are checked in these twosubproblems. Scenarios are dynamically clustered during the subproblem solution process through the multiparametric programming with respect to the solution of the master problem. In other words, multiple scenariosare clustered into several representative scenarios after the subproblem is solved, and the Benders cut obtainedby the representative scenario is generated for the master problem. Different from the conventional stochasticunit commitment, the proposed approach integrates scenario clustering into the Benders decomposition solutionprocess. Such a clustering approach could accurately cluster representative scenarios that have impacts on theunit commitment. The proposed method is tested on a 6-bus system and the modified IEEE 118-bus system.Numerical results illustrate the effectiveness of the proposed method in clustering scenarios. Compared withthe conventional clustering method, the proposed method can accurately select representative scenarios whilemitigating computational burden, thus guaranteeing the robustness of unit commitment.

Neurocircuit regeneration by extracellular matrix reprogramming

The brain's extracellular matrix(ECM),which is comprised of protein and glycosaminoglycan(GAG)scaffolds,constitutes 20%-40% of the human brain and is considered one of the largest influencers on brain cell functioning(Soles et al.,2023).Synthesized by neural and glial cells,the brain's ECM regulates a myriad of homeostatic cellular processes,including neuronal plasticity and firing(Miyata et al.,2012),cation buffering(Moraws ki et al.,2015),and glia-neuron interactions(Anderson et al.,2016).Considering the diversity of functions,dynamic remodeling of the brain's ECM indicates that this understudied medium is an active participant in both normal physiology and neurological diseases.

Transient Stability Preventive Control of Wind Farm Connected Power System Considering the Uncertainty

To address uncertainty as well as transient stability constraints simultaneously in the preventive control of windfarm systems, a novel three-stage optimization strategy is established in this paper. In the first stage, the probabilisticmulti-objective particle swarm optimization based on the point estimate method is employed to cope with thestochastic factors. The transient security region of the system is accurately ensured by the interior point methodin the second stage. Finally, the verification of the final optimal objectives and satisfied constraints are enforcedin the last stage. Furthermore, the proposed strategy is a general framework that can combine other optimizationalgorithms. The proposed methodology is tested on the modified WSCC 9-bus system and the New England 39-bussystem. The results verify the feasibility of the method.

A Home Automation Control System Based on LabVIEW

The objective of this project is to explore the possibility of using X-10 and LabVIEW to control the device in the house. Based on the serial port communication of LabVIEW, the X-10 module can be programmed by using the X-10 commands. Through the power line, all the devices connected to the socket will be controlled. Without replacing the existing wire, it must be an easy control system for the user who has no experience in electronics or communication engineering. Actually, this is a quite practical X-10 home automation system.

A Dimensional Reduction Approach Based on Essential Constraints in Linear Programming

This paper presents a new dimension reduction strategy for medium and large-scale linear programming problems. The proposed method uses a subset of the original constraints and combines two algorithms: the weighted average and the cosine simplex algorithm. The first approach identifies binding constraints by using the weighted average of each constraint, whereas the second algorithm is based on the cosine similarity between the vector of the objective function and the constraints. These two approaches are complementary, and when used together, they locate the essential subset of initial constraints required for solving medium and large-scale linear programming problems. After reducing the dimension of the linear programming problem using the subset of the essential constraints, the solution method can be chosen from any suitable method for linear programming. The proposed approach was applied to a set of well-known benchmarks as well as more than 2000 random medium and large-scale linear programming problems. The results are promising, indicating that the new approach contributes to the reduction of both the size of the problems and the total number of iterations required. A tree-based classification model also confirmed the need for combining the two approaches. A detailed numerical example, the general numerical results, and the statistical analysis for the decision tree procedure are presented.

Optimization Techniques for GPU-Based Parallel Programming Models in High-Performance

This study embarks on a comprehensive examination of optimization techniques within GPU-based parallel programming models,pivotal for advancing high-performance computing(HPC).Emphasizing the transition of GPUs from graphic-centric processors to versatile computing units,it delves into the nuanced optimization of memory access,thread management,algorithmic design,and data structures.These optimizations are critical for exploiting the parallel processing capabilities of GPUs,addressingboth the theoretical frameworks and practical implementations.By integrating advanced strategies such as memory coalescing,dynamic scheduling,and parallel algorithmic transformations,this research aims to significantly elevate computational efficiency and throughput.The findings underscore the potential of optimized GPU programming to revolutionize computational tasks across various domains,highlighting a pathway towards achieving unparalleled processing power and efficiency in HPC environments.The paper not only contributes to the academic discourse on GPU optimization but also provides actionable insights for developers,fostering advancements in computational sciences and technology.

Suggestions on Improving China’s Breakthrough Therapy Designation System

Objective To analyze the characteristics of breakthrough therapy designation(BTD)and its implementation in China,and to provide reference for the optimization of BTD system.Methods A comparative research method was used to study the content and implementation effect of BTD system in China and the relevant policies and implementation of the same procedures of drug regulatory authorities in the United States,Japan and the European Union.Then,the differences in policies and implementation results among these countries were analyzed to provide suggestions for the implementation and optimization of this system in China.Results and Conclusion China’s BTD system is implemented late and a small number of drugs has been approved.At the same time,there are problems such as insufficient guidance and communication from the agency to applicants,a broad application condition,single review mode,and lack of full-time personnel.Both the agencies and the applicants have limited experience due to the short implementation time of BTD system in China.There are still some problems despite we have learned a lot from the experience of other drug regulatory agencies.Therefore,based on our national conditions,we should strengthen the guidance of evaluation agency to applicants,optimize the eligibility criteria of BTD system,introduce the rolling review,and increase the number of professional liaisons,which can accelerate the development and marketing process of drugs with obvious clinical value,and finally to address unmet medical need.

Event-based performance guaranteed tracking control for constrained nonlinear system via adaptive dynamic programming method

An optimal tracking control problem for a class of nonlinear systems with guaranteed performance and asymmetric input constraints is discussed in this paper.The control policy is implemented by adaptive dynamic programming(ADP)algorithm under two event-based triggering mechanisms.It is often challenging to design an optimal control law due to the system deviation caused by asymmetric input constraints.First,a prescribed performance control technique is employed to guarantee the tracking errors within predetermined boundaries.Subsequently,considering the asymmetric input constraints,a discounted non-quadratic cost function is introduced.Moreover,in order to reduce controller updates,an event-triggered control law is developed for ADP algorithm.After that,to further simplify the complexity of controller design,this work is extended to a self-triggered case for relaxing the need for continuous signal monitoring by hardware devices.By employing the Lyapunov method,the uniform ultimate boundedness of all signals is proved to be guaranteed.Finally,a simulation example on a mass–spring–damper system subject to asymmetric input constraints is provided to validate the effectiveness of the proposed control scheme.

Coordinated planning for flexible interconnection and energy storage system in low-voltage distribution networks to improve the accommodation capacity of photovoltaic

The increasing proportion of distributed photovoltaics(DPVs)and electric vehicle charging stations in low-voltage distribution networks(LVDNs)has resulted in challenges such as distribution transformer overloads and voltage violations.To address these problems,we propose a coordinated planning method for flexible interconnections and energy storage systems(ESSs)to improve the accommodation capacity of DPVs.First,the power-transfer characteristics of flexible interconnection and ESSs are analyzed.The equipment costs of the voltage source converters(VSCs)and ESSs are also analyzed comprehensively,considering the differences in installation and maintenance costs for different installation locations.Second,a bilevel programming model is established to minimize the annual comprehensive cost and yearly total PV curtailment capacity.Within this framework,the upper-level model optimizes the installation locations and capacities of the VSCs and ESSs,whereas the lower-level model optimizes the operating power of the VSCs and ESSs.The proposed model is solved using a non-dominated sorting genetic algorithm with an elite strategy(NSGA-II).The effectiveness of the proposed planning method is validated through an actual LVDN scenario,which demonstrates its advantages in enhancing PV accommodation capacity.In addition,the economic benefits of various planning schemes with different flexible interconnection topologies and different PV grid-connected forms are quantitatively analyzed,demonstrating the adaptability of the proposed coordinated planning method.

Sequential quadratic programming-based non-cooperative target distributed hybrid processing optimization method

The distributed hybrid processing optimization problem of non-cooperative targets is an important research direction for future networked air-defense and anti-missile firepower systems. In this paper, the air-defense anti-missile targets defense problem is abstracted as a nonconvex constrained combinatorial optimization problem with the optimization objective of maximizing the degree of contribution of the processing scheme to non-cooperative targets, and the constraints mainly consider geographical conditions and anti-missile equipment resources. The grid discretization concept is used to partition the defense area into network nodes, and the overall defense strategy scheme is described as a nonlinear programming problem to solve the minimum defense cost within the maximum defense capability of the defense system network. In the solution of the minimum defense cost problem, the processing scheme, equipment coverage capability, constraints and node cost requirements are characterized, then a nonlinear mathematical model of the non-cooperative target distributed hybrid processing optimization problem is established, and a local optimal solution based on the sequential quadratic programming algorithm is constructed, and the optimal firepower processing scheme is given by using the sequential quadratic programming method containing non-convex quadratic equations and inequality constraints. Finally, the effectiveness of the proposed method is verified by simulation examples.

Policy Iteration for Optimal Control of Discrete-Time Time-Varying Nonlinear Systems

Aimed at infinite horizon optimal control problems of discrete time-varying nonlinear systems,in this paper,a new iterative adaptive dynamic programming algorithm,which is the discrete-time time-varying policy iteration(DTTV)algorithm,is developed.The iterative control law is designed to update the iterative value function which approximates the index function of optimal performance.The admissibility of the iterative control law is analyzed.The results show that the iterative value function is non-increasingly convergent to the Bellman-equation optimal solution.To implement the algorithm,neural networks are employed and a new implementation structure is established,which avoids solving the generalized Bellman equation in each iteration.Finally,the optimal control laws for torsional pendulum and inverted pendulum systems are obtained by using the DTTV policy iteration algorithm,where the mass and pendulum bar length are permitted to be time-varying parameters.The effectiveness of the developed method is illustrated by numerical results and comparisons.

A true triaxial strength criterion for rocks by gene expression programming

Rock strength is a crucial factor to consider when designing and constructing underground projects.This study utilizes a gene expression programming(GEP)algorithm-based model to predict the true triaxial strength of rocks,taking into account the influence of rock genesis on their mechanical behavior during the model building process.A true triaxial strength criterion based on the GEP model for igneous,metamorphic and magmatic rocks was obtained by training the model using collected data.Compared to the modified Weibols-Cook criterion,the modified Mohr-Coulomb criterion,and the modified Lade criterion,the strength criterion based on the GEP model exhibits superior prediction accuracy performance.The strength criterion based on the GEP model has better performance in R2,RMSE and MAPE for the data set used in this study.Furthermore,the strength criterion based on the GEP model shows greater stability in predicting the true triaxial strength of rocks across different types.Compared to the existing strength criterion based on the genetic programming(GP)model,the proposed criterion based on GEP model achieves more accurate predictions of the variation of true triaxial strength(s1)with intermediate principal stress(s2).Finally,based on the Sobol sensitivity analysis technique,the effects of the parameters of the three obtained strength criteria on the true triaxial strength of the rock are analysed.In general,the proposed strength criterion exhibits superior performance in terms of both accuracy and stability of prediction results.

Hepatic arterial infusion chemotherapy with anti-angiogenesis agents and immune checkpoint inhibitors for unresectable hepatocellular carcinoma and meta-analysis

BACKGROUND Hepatic arterial infusion chemotherapy(HAIC)has been proven to be an ideal choice for treating unresectable hepatocellular carcinoma(uHCC).HAIC-based treatment showed great potential for treating uHCC.However,large-scale studies on HAIC-based treatments and meta-analyses of first-line treatments for uHCC are lacking.AIM To investigate better first-line treatment options for uHCC and to assess the safety and efficacy of HAIC combined with angiogenesis inhibitors,programmed cell death of protein 1(PD-1)and its ligand(PD-L1)blockers(triple therapy)under real-world conditions.METHODS Several electronic databases were searched to identify eligible randomized controlled trials for this meta-analysis.Study-level pooled analyses of hazard ratios(HRs)and odds ratios(ORs)were performed.This was a retrospective single-center study involving 442 patients with uHCC who received triple therapy or angiogenesis inhibitors plus PD-1/PD-L1 blockades(AIPB)at Sun Yat-sen University Cancer Center from January 2018 to April 2023.Propensity score matching(PSM)was performed to balance the bias between the groups.The Kaplan-Meier method and cox regression were used to analyse the survival data,and the log-rank test was used to compare the suvival time between the groups.RESULTS A total of 13 randomized controlled trials were included.HAIC alone and in combination with sorafenib were found to be effective treatments(P values for ORs:HAIC,0.95;for HRs:HAIC+sorafenib,0.04).After PSM,176 HCC patients were included in the analysis.The triple therapy group(n=88)had a longer median overall survival than the AIPB group(n=88)(31.6 months vs 14.6 months,P<0.001)and a greater incidence of adverse events(94.3%vs 75.4%,P<0.001).CONCLUSION This meta-analysis suggests that HAIC-based treatments are likely to be the best choice for uHCC.Our findings confirm that triple therapy is more effective for uHCC patients than AIPB.

Designing and Optimization of an Off-line Programming System for Robotic Belt Grinding Process

Off-line programming （OLP） system becomes one of the most important programming modules for the robotic belt grinding process, however there lacks research on increasing the grinding dexterous space depending on the OLP system. A new type of grinding robot and a novel robotic belt grinding workcell are forwarded, and their features are briefly introduced. An open and object-oriented off-line programming system is developed for this robotic belt grinding system. The parameters of the trimmed surface are read from the initial graphics exchange specification （IGES） file of the CAD model of the workpiece. The deBoor-Cox basis function is used to sample the grinding target with local contact frame on the workpiece. The numerical formula of inverse kinematics is set up based on Newton＇s iterative procedure, to calculate the grinding robot configurations corresponding to the grinding targets. After the grinding path is obtained, the OLP system turns to be more effective than the teach-by-showing system. In order to improve the grinding workspace, an optimization algorithm for dynamic tool frame is proposed and performed on the special robotic belt grinding system. The initial tool frame and the interval of neighboring tool frames are defined as the preparation of the algorithm. An optimized tool local frame can be selected to grind the complex surface for a maximum dexterity index of the robot. Under the optimization algorithm, a simulation of grinding a vane is included and comparison of grinding workspace is done before and after the tool frame optimization. By the algorithm, the grinding workspace can be enlarged. Moreover the dynamic tool frame can be considered to add one degree-of-freedom to the grinding kinematical chain, which provides the theoretical support for the improvement of robotic dexterity for the complex surface grinding.

A Stochastic Programming Strategy in Microgrid Cyber Physical Energy System for Energy Optimal Operation

This paper focuses on the energy optimal operation problem of microgrids (MGs) under stochastic environment. The deterministic method of MGs operation is often uneconomical because it fails to consider the high randomness of unconventional energy resources. Therefore, it is necessary to develop a novel operation approach combining the uncertainty in the physical world with modeling strategy in the cyber system. This paper proposes an energy scheduling optimization strategy based on stochastic programming model by considering the uncertainty in MGs. The goal is to minimize the expected operation cost of MGs. The uncertainties are modeled based on autoregressive moving average (ARMA) model to expose the effects of physical world on cyber world. Through the comparison of the simulation results with deterministic method, it is shown that the effectiveness and robustness of proposed stochastic energy scheduling optimization strategy for MGs are valid. © 2014 Chinese Association of Automation.