Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Adaptive Optimal Discrete-Time Output-Feedback Using an Internal Model Principle and Adaptive Dynamic Programming

In order to address the output feedback issue for linear discrete-time systems, this work suggests a brand-new adaptive dynamic programming(ADP) technique based on the internal model principle(IMP). The proposed method, termed as IMP-ADP, does not require complete state feedback-merely the measurement of input and output data. More specifically, based on the IMP, the output control problem can first be converted into a stabilization problem. We then design an observer to reproduce the full state of the system by measuring the inputs and outputs. Moreover, this technique includes both a policy iteration algorithm and a value iteration algorithm to determine the optimal feedback gain without using a dynamic system model. It is important that with this concept one does not need to solve the regulator equation. Finally, this control method was tested on an inverter system of grid-connected LCLs to demonstrate that the proposed method provides the desired performance in terms of both tracking and disturbance rejection.

Improved Unit Commitment with Accurate Dynamic Scenarios Clustering Based on Multi-Parametric Programming and Benders Decomposition

Stochastic unit commitment is one of the most powerful methods to address uncertainty. However, the existingscenario clustering technique for stochastic unit commitment cannot accurately select representative scenarios,which threatens the robustness of stochastic unit commitment and hinders its application. This paper providesa stochastic unit commitment with dynamic scenario clustering based on multi-parametric programming andBenders decomposition. The stochastic unit commitment is solved via the Benders decomposition, which decouplesthe primal problem into the master problem and two types of subproblems. In the master problem, the committedgenerator is determined, while the feasibility and optimality of generator output are checked in these twosubproblems. Scenarios are dynamically clustered during the subproblem solution process through the multiparametric programming with respect to the solution of the master problem. In other words, multiple scenariosare clustered into several representative scenarios after the subproblem is solved, and the Benders cut obtainedby the representative scenario is generated for the master problem. Different from the conventional stochasticunit commitment, the proposed approach integrates scenario clustering into the Benders decomposition solutionprocess. Such a clustering approach could accurately cluster representative scenarios that have impacts on theunit commitment. The proposed method is tested on a 6-bus system and the modified IEEE 118-bus system.Numerical results illustrate the effectiveness of the proposed method in clustering scenarios. Compared withthe conventional clustering method, the proposed method can accurately select representative scenarios whilemitigating computational burden, thus guaranteeing the robustness of unit commitment.

Glia-to-neuron reprogramming to the rescue?

Over the last two decades,the dogma that cell fate is immutable has been increasingly challenged,with important implications for regenerative medicine.The brea kth rough discovery that induced pluripotent stem cells could be generated from adult mouse fibroblasts is powerful proof that cell fate can be changed.An exciting extension of the discovery of cell fate impermanence is the direct cellular reprogram ming hypothesis-that terminally differentiated cells can be reprogrammed into other adult cell fates without first passing through a stem cell state.

Neurocircuit regeneration by extracellular matrix reprogramming

The brain's extracellular matrix(ECM),which is comprised of protein and glycosaminoglycan(GAG)scaffolds,constitutes 20%-40% of the human brain and is considered one of the largest influencers on brain cell functioning(Soles et al.,2023).Synthesized by neural and glial cells,the brain's ECM regulates a myriad of homeostatic cellular processes,including neuronal plasticity and firing(Miyata et al.,2012),cation buffering(Moraws ki et al.,2015),and glia-neuron interactions(Anderson et al.,2016).Considering the diversity of functions,dynamic remodeling of the brain's ECM indicates that this understudied medium is an active participant in both normal physiology and neurological diseases.

Apples to oranges:environmentally derived,dynamic regulation of serotonin neuron subpopulations in adulthood?

Traumatic brain injury(TBI)is a public health problem with an undue economic burden that impacts nearly every age,ethnic,and gender group across the globe(Capizzi et al.,2020).TBIs are often sustained during a dynamic range of exposures to energetic environmental forces and as such outcomes are typically heterogeneous regarding severity and pathology(Capizzi et al.,2020).

Nanograting‑Based Dynamic Structural Colors Using Heterogeneous Materials

Dynamic structuralcolors can change in response todifferent environmental stimuli.This ability remains effectiveeven when the size of the speciesresponsible for the structural coloris reduced to a few micrometers,providing a promising sensingmechanism for solving microenvironmentalsensing problems inmicro-robotics and microfluidics.However, the lack of dynamicstructural colors that can encoderapidly, easily integrate, and accuratelyreflect changes in physical quantities hinders their use in microscale sensing applications. Herein, we present a 2.5-dimensionaldynamic structural color based on nanogratings of heterogeneous materials, which were obtained by interweaving a pH-responsive hydrogelwith an IP-L photoresist. Transverse gratings printed with pH-responsive hydrogels elongated the period of longitudinal grating in the swollenstate, resulting in pH-tuned structural colors at a 45° incidence. Moreover, the patterned encoding and array printing of dynamic structuralcolors were achieved using grayscale stripe images to accurately encode the periods and heights of the nanogrid structures. Overall, dynamicstructural color networks exhibit promising potential for applications in information encryption and in situ sensing for microfluidic chips.

A Rapid Adaptation Approach for Dynamic Air‑Writing Recognition Using Wearable Wristbands with Self‑Supervised Contrastive Learning

Wearable wristband systems leverage deep learning to revolutionize hand gesture recognition in daily activities.Unlike existing approaches that often focus on static gestures and require extensive labeled data,the proposed wearable wristband with selfsupervised contrastive learning excels at dynamic motion tracking and adapts rapidly across multiple scenarios.It features a four-channel sensing array composed of an ionic hydrogel with hierarchical microcone structures and ultrathin flexible electrodes,resulting in high-sensitivity capacitance output.Through wireless transmission from a Wi-Fi module,the proposed algorithm learns latent features from the unlabeled signals of random wrist movements.Remarkably,only few-shot labeled data are sufficient for fine-tuning the model,enabling rapid adaptation to various tasks.The system achieves a high accuracy of 94.9%in different scenarios,including the prediction of eight-direction commands,and air-writing of all numbers and letters.The proposed method facilitates smooth transitions between multiple tasks without the need for modifying the structure or undergoing extensive task-specific training.Its utility has been further extended to enhance human–machine interaction over digital platforms,such as game controls,calculators,and three-language login systems,offering users a natural and intuitive way of communication.

Efficient and Stable Perovskite Solar Cells and Modules Enabled by Tailoring Additive Distribution According to the Film Growth Dynamics

Gas quenching and vacuum quenching process are widely applied to accelerate solvent volatilization to induce nucleation of perovskites in blade-coating method.In this work,we found these two pre-crystallization processes lead to different order of crystallization dynamics within the perovskite thin film,resulting in the differences of additive distribution.We then tailor-designed an additive molecule named 1,3-bis(4-methoxyphenyl)thiourea to obtain films with fewer defects and holes at the buried interface,and prepared perovskite solar cells with a certified efficiency of 23.75%.Furthermore,this work also demonstrates an efficiency of 20.18%for the large-area perovskite solar module(PSM)with an aperture area of 60.84 cm^(2).The PSM possesses remarkable continuous operation stability for maximum power point tracking of T_(90)>1000 h in ambient air.

Dynamic Multi-Graph Spatio-Temporal Graph Traffic Flow Prediction in Bangkok:An Application of a Continuous Convolutional Neural Network

The ability to accurately predict urban traffic flows is crucial for optimising city operations.Consequently,various methods for forecasting urban traffic have been developed,focusing on analysing historical data to understand complex mobility patterns.Deep learning techniques,such as graph neural networks(GNNs),are popular for their ability to capture spatio-temporal dependencies.However,these models often become overly complex due to the large number of hyper-parameters involved.In this study,we introduce Dynamic Multi-Graph Spatial-Temporal Graph Neural Ordinary Differential Equation Networks(DMST-GNODE),a framework based on ordinary differential equations(ODEs)that autonomously discovers effective spatial-temporal graph neural network(STGNN)architectures for traffic prediction tasks.The comparative analysis of DMST-GNODE and baseline models indicates that DMST-GNODE model demonstrates superior performance across multiple datasets,consistently achieving the lowest Root Mean Square Error(RMSE)and Mean Absolute Error(MAE)values,alongside the highest accuracy.On the BKK(Bangkok)dataset,it outperformed other models with an RMSE of 3.3165 and an accuracy of 0.9367 for a 20-min interval,maintaining this trend across 40 and 60 min.Similarly,on the PeMS08 dataset,DMST-GNODE achieved the best performance with an RMSE of 19.4863 and an accuracy of 0.9377 at 20 min,demonstrating its effectiveness over longer periods.The Los_Loop dataset results further emphasise this model’s advantage,with an RMSE of 3.3422 and an accuracy of 0.7643 at 20 min,consistently maintaining superiority across all time intervals.These numerical highlights indicate that DMST-GNODE not only outperforms baseline models but also achieves higher accuracy and lower errors across different time intervals and datasets.

Stability Prediction in Smart Grid Using PSO Optimized XGBoost Algorithm with Dynamic Inertia Weight Updation

Prediction of stability in SG(Smart Grid)is essential in maintaining consistency and reliability of power supply in grid infrastructure.Analyzing the fluctuations in power generation and consumption patterns of smart cities assists in effectively managing continuous power supply in the grid.It also possesses a better impact on averting overloading and permitting effective energy storage.Even though many traditional techniques have predicted the consumption rate for preserving stability,enhancement is required in prediction measures with minimized loss.To overcome the complications in existing studies,this paper intends to predict stability from the smart grid stability prediction dataset using machine learning algorithms.To accomplish this,pre-processing is performed initially to handle missing values since it develops biased models when missing values are mishandled and performs feature scaling to normalize independent data features.Then,the pre-processed data are taken for training and testing.Following that,the regression process is performed using Modified PSO(Particle Swarm Optimization)optimized XGBoost Technique with dynamic inertia weight update,which analyses variables like gamma(G),reaction time(tau1–tau4),and power balance(p1–p4)for providing effective future stability in SG.Since PSO attains optimal solution by adjusting position through dynamic inertial weights,it is integrated with XGBoost due to its scalability and faster computational speed characteristics.The hyperparameters of XGBoost are fine-tuned in the training process for achieving promising outcomes on prediction.Regression results are measured through evaluation metrics such as MSE(Mean Square Error)of 0.011312781,MAE(Mean Absolute Error)of 0.008596322,and RMSE(Root Mean Square Error)of 0.010636156 and MAPE(Mean Absolute Percentage Error)value of 0.0052 which determine the efficacy of the system.

Efficient and Stable Photoassisted Lithium‑Ion Battery Enabled by Photocathode with Synergistically Boosted Carriers Dynamics

Efficient and stable photocathodes with versatility are of significance in photoassisted lithium-ion batteries(PLIBs),while there is always a request on fast carrier transport in electrochemical active photocathodes.Present work proposes a general approach of creating bulk heterojunction to boost the carrier mobility of photocathodes by simply laser assisted embedding of plasmonic nanocrystals.When employed in PLIBs,it was found effective for synchronously enhanced photocharge separation and transport in light charging process.Additionally,experimental photon spectroscopy,finite difference time domain method simulation and theoretical analyses demonstrate that the improved carrier dynamics are driven by the plasmonic-induced hot electron injection from metal to TiO_(2),as well as the enhanced conductivity in TiO2 matrix due to the formation of oxygen vacancies after Schottky contact.Benefiting from these merits,several benchmark values in performance of TiO2-based photocathode applied in PLIBs are set,including the capacity of 276 mAh g^(−1)at 0.2 A g^(−1)under illumination,photoconversion efficiency of 1.276%at 3 A g^(−1),less capacity and Columbic efficiency loss even through 200 cycles.These results exemplify the potential of the bulk heterojunction strategy in developing highly efficient and stable photoassisted energy storage systems.

Emerging structures and dynamic mechanisms ofγ-secretase for Alzheimer’s disease

γ-Secretase,called“the proteasome of the membrane,”is a membrane-embedded protease complex that cleaves 150+peptide substrates with central roles in biology and medicine,including amyloid precursor protein and the Notch family of cell-surface receptors.Mutations inγ-secretase and amyloid precursor protein lead to early-onset familial Alzheimer’s disease.γ-Secretase has thus served as a critical drug target for treating familial Alzheimer’s disease and the more common late-onset Alzheimer’s disease as well.However,critical gaps remain in understanding the mechanisms of processive proteolysis of substrates,the effects of familial Alzheimer’s disease mutations,and allosteric modulation of substrate cleavage byγ-secretase.In this review,we focus on recent studies of structural dynamic mechanisms ofγ-secretase.Different mechanisms,including the“Fit-Stay-Trim,”“Sliding-Unwinding,”and“Tilting-Unwinding,”have been proposed for substrate proteolysis of amyloid precursor protein byγ-secretase based on all-atom molecular dynamics simulations.While an incorrect registry of the Notch1 substrate was identified in the cryo-electron microscopy structure of Notch1-boundγ-secretase,molecular dynamics simulations on a resolved model of Notch1-boundγ-secretase that was reconstructed using the amyloid precursor protein-boundγ-secretase as a template successfully capturedγ-secretase activation for proper cleavages of both wildtype and mutant Notch,being consistent with biochemical experimental findings.The approach could be potentially applied to decipher the processing mechanisms of various substrates byγ-secretase.In addition,controversy over the effects of familial Alzheimer’s disease mutations,particularly the issue of whether they stabilize or destabilizeγ-secretase-substrate complexes,is discussed.Finally,an outlook is provided for future studies ofγ-secretase,including pathways of substrate binding and product release,effects of modulators on familial Alzheimer’s disease mutations of theγ-secretase-substrate complexes.Comprehensive understanding of the functional mechanisms ofγ-secretase will greatly facilitate the rational design of effective drug molecules for treating familial Alzheimer’s disease and perhaps Alzheimer’s disease in general.

Dynamic Interaction Analysis of Coupled Axial-Torsional-Lateral Mechanical Vibrations in Rotary Drilling Systems

Maintaining the integrity and longevity of structures is essential in many industries,such as aerospace,nuclear,and petroleum.To achieve the cost-effectiveness of large-scale systems in petroleum drilling,a strong emphasis on structural durability and monitoring is required.This study focuses on the mechanical vibrations that occur in rotary drilling systems,which have a substantial impact on the structural integrity of drilling equipment.The study specifically investigates axial,torsional,and lateral vibrations,which might lead to negative consequences such as bit-bounce,chaotic whirling,and high-frequency stick-slip.These events not only hinder the efficiency of drilling but also lead to exhaustion and harm to the system’s components since they are difficult to be detected and controlled in real time.The study investigates the dynamic interactions of these vibrations,specifically in their high-frequency modes,usingfield data obtained from measurement while drilling.Thefindings have demonstrated the effect of strong coupling between the high-frequency modes of these vibrations on drilling sys-tem performance.The obtained results highlight the importance of considering the interconnected impacts of these vibrations when designing and implementing robust control systems.Therefore,integrating these compo-nents can increase the durability of drill bits and drill strings,as well as improve the ability to monitor and detect damage.Moreover,by exploiting thesefindings,the assessment of structural resilience in rotary drilling systems can be enhanced.Furthermore,the study demonstrates the capacity of structural health monitoring to improve the quality,dependability,and efficiency of rotary drilling systems in the petroleum industry.

Shrek:a dynamic object-oriented programming language

From a perspective of theoretical study, there are some faults in the models of the existing object-oriented programming languages. For example, C# does not support metaclasses, the primitive types of Java and C# are not objects, etc. So, this paper designs a programming language, Shrek, which integrates many language features and constructions in a compact and consistent model. The Shrek language is a class-based purely object-oriented language. It has a dynamical strong type system, and adopts a single-inheritance mechanism with Mixin as its complement. It has a consistent class instantiation and inheritance structure, and the ability of intercessive structural computational reflection, which enables it to support safe metaclass programming. It also supports multi-thread programming and automatic garbage collection, and enforces its expressive power by adopting a native method mechanism. The prototype system of the Shrek language is implemented and anticipated design goals are achieved.

Parallel Control for Optimal Tracking via Adaptive Dynamic Programming

This paper studies the problem of optimal parallel tracking control for continuous-time general nonlinear systems.Unlike existing optimal state feedback control,the control input of the optimal parallel control is introduced into the feedback system.However,due to the introduction of control input into the feedback system,the optimal state feedback control methods can not be applied directly.To address this problem,an augmented system and an augmented performance index function are proposed firstly.Thus,the general nonlinear system is transformed into an affine nonlinear system.The difference between the optimal parallel control and the optimal state feedback control is analyzed theoretically.It is proven that the optimal parallel control with the augmented performance index function can be seen as the suboptimal state feedback control with the traditional performance index function.Moreover,an adaptive dynamic programming(ADP)technique is utilized to implement the optimal parallel tracking control using a critic neural network(NN)to approximate the value function online.The stability analysis of the closed-loop system is performed using the Lyapunov theory,and the tracking error and NN weights errors are uniformly ultimately bounded(UUB).Also,the optimal parallel controller guarantees the continuity of the control input under the circumstance that there are finite jump discontinuities in the reference signals.Finally,the effectiveness of the developed optimal parallel control method is verified in two cases.

Residential Energy Scheduling for Variable Weather Solar Energy Based on Adaptive Dynamic Programming

The residential energy scheduling of solar energy is an important research area of smart grid. On the demand side, factors such as household loads, storage batteries, the outside public utility grid and renewable energy resources, are combined together as a nonlinear, time-varying, indefinite and complex system, which is difficult to manage or optimize. Many nations have already applied the residential real-time pricing to balance the burden on their grid. In order to enhance electricity efficiency of the residential micro grid, this paper presents an action dependent heuristic dynamic programming(ADHDP) method to solve the residential energy scheduling problem. The highlights of this paper are listed below. First,the weather-type classification is adopted to establish three types of programming models based on the features of the solar energy. In addition, the priorities of different energy resources are set to reduce the loss of electrical energy transmissions.Second, three ADHDP-based neural networks, which can update themselves during applications, are designed to manage the flows of electricity. Third, simulation results show that the proposed scheduling method has effectively reduced the total electricity cost and improved load balancing process. The comparison with the particle swarm optimization algorithm further proves that the present method has a promising effect on energy management to save cost.

Approach of service recovery decision-making based on Bellman dynamic programming

Based on service-oriented architecture（SOA）,a Bellman-dynamic-programming-based approach of service recovery decision-making is proposed to make valid recovery decisions.Both the attribute and the process of services in the controllable distributed information system are analyzed as the preparatory work.Using the idea of service composition as a reference,the approach translates the recovery decision-making into a planning problem regarding artificial intelligence （AI） through two steps.The first is the self-organization based on a logical view of the network,and the second is the definition of evaluation standards.Applying Bellman dynamic programming to solve the planning problem,the approach offers timely emergency response and optimal recovery source selection,meeting multiple QoS （quality of service）requirements.Experimental results demonstrate the rationality and optimality of the approach,and the theoretical analysis of its computational complexity and the comparison with conventional methods exhibit its high efficiency.

Using approximate dynamic programming for multi-ESM scheduling to track ground moving targets

This paper researches the adaptive scheduling problem of multiple electronic support measures(multi-ESM) in a ground moving radar targets tracking application. It is a sequential decision-making problem in uncertain environment. For adaptive selection of appropriate ESMs, we generalize an approximate dynamic programming(ADP) framework to the dynamic case. We define the environment model and agent model, respectively. To handle the partially observable challenge, we apply the unsented Kalman filter(UKF) algorithm for belief state estimation. To reduce the computational burden, a simulation-based approach rollout with a redesigned base policy is proposed to approximate the long-term cumulative reward. Meanwhile, Monte Carlo sampling is combined into the rollout to estimate the expectation of the rewards. The experiments indicate that our method outperforms other strategies due to its better performance in larger-scale problems.

Approximate Dynamic Programming for Stochastic Resource Allocation Problems

A stochastic resource allocation model, based on the principles of Markov decision processes(MDPs), is proposed in this paper. In particular, a general-purpose framework is developed, which takes into account resource requests for both instant and future needs. The considered framework can handle two types of reservations(i.e., specified and unspecified time interval reservation requests), and implement an overbooking business strategy to further increase business revenues. The resulting dynamic pricing problems can be regarded as sequential decision-making problems under uncertainty, which is solved by means of stochastic dynamic programming(DP) based algorithms. In this regard, Bellman’s backward principle of optimality is exploited in order to provide all the implementation mechanisms for the proposed reservation pricing algorithm. The curse of dimensionality, as the inevitable issue of the DP both for instant resource requests and future resource reservations,occurs. In particular, an approximate dynamic programming(ADP) technique based on linear function approximations is applied to solve such scalability issues. Several examples are provided to show the effectiveness of the proposed approach.