Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Molecular dynamics simulation of the flow mechanism of shear-thinning fluids in a microchannel

Shear-thinning fluids have been widely used in microfluidic systems,but their internal flow mechanism is still unclear.Therefore,in this paper,molecular dynamics simulations are used to study the laminar flow of shear-thinning fluid in a microchannel.We validated the feasibility of our simulation method by evaluating the mean square displacement and Reynolds number of the solution layers.The results show that the change rule of the fluid system's velocity profile and interaction energy can reflect the shear-thinning characteristics of the fluids.The velocity profile resembles a top-hat shape,intensifying as the fluid's power law index decreases.The interaction energy between the wall and the fluid decreases gradually with increasing velocity,and a high concentration of non-Newtonian fluid reaches a plateau sooner.Moreover,the velocity profile of the fluid is related to the molecule number density distribution and their values are inversely proportional.By analyzing the radial distribution function,we found that the hydrogen bonds between solute and water molecules weaken with the increase in velocity.This observation offers an explanation for the shear-thinning phenomenon of the non-Newtonian flow from a micro perspective.

Temperature effect on nanotwinned Ni under nanoindentation using molecular dynamic simulation

Temperature effect on atomic deformation of nanotwinned Ni (nt-Ni) under localized nanoindentation is investigated in comparison with nanocrystalline Ni (nc-Ni) through molecular simulation.The nt-Ni exhibits enhanced critical load and hardness compared to nc-Ni,where perfect,stair-rod and Shockley dislocations are activated at (111),(111) and (111) slip planes in nt-Ni compared to only SSockley dislocation nucleation at (111) and (111) slip planes of nc-Ni.The nt-Ni exhibits a less significant indentation size effect in comparison with nc-Ni due to the dislocation slips hindrance of the twin boundary.The atomic deformation associated with the indentation size effect is investigated during dislocation transmission.Different from the decreasing partial slips parallel to the indenter surface in nc-Ni with increasing temperature,the temperaturedependent atomic deformation of nt-Ni is closely related to the twin boundary:from the partial slips parallel to the twin boundary (~10 K),to increased confined layer slips and decreased twin migration(300 K–600 K),to decreased confined layer slips and increased dislocation interaction of dislocation pinning and dissociation (900 K–1200 K).Dislocation density and atomic structure types through quantitative analysis are implemented to further reveal the above-mentioned dislocation motion and atomic structure alteration.Our study is helpful for understanding the temperature-dependent plasticity of twin boundary in nanotwinned materials.

Investigation of Projectile Impact Behaviors of Graphene Aerogel Using Molecular Dynamics Simulations

Graphene aerogel(GA),as a novel solid material,has shown great potential in engineering applications due to its unique mechanical properties.In this study,the mechanical performance of GA under high-velocity projectile impacts is thoroughly investigated using full-atomic molecular dynamics(MD)simulations.The study results show that the porous structure and density are key factors determining the mechanical response of GA under impact loading.Specifically,the impact-induced penetration of the projectile leads to the collapse of the pore structure,causing stretching and subsequent rupture of covalent bonds in graphene sheets.Moreover,the effects of temperature on the mechanical performance of GA have been proven to be minimal,thereby highlighting the mechanical stability of GA over a wide range of temperatures.Finally,the energy absorption density(EAD)and energy absorption efficiency(EAE)metrics are adopted to assess the energy absorption capacity of GA during projectile penetration.The research findings of this work demonstrate the significant potential of GA for energy absorption applications.

Study of the Relationship Between New Ionic Interaction Parameters and Salt Solubility in Electrolyte Solutions Based on Molecular Dynamics Simulation

Studying the relationship between ionic interactions and salt solubility in seawater has implications for seawater desalination and mineral extraction.In this paper,a new method of expressing ion-to-ion interaction is proposed by using molecular dynamics simulation,and the relationship between ion-to-ion interaction and salt solubility in a simulated seawater water-salt system is investigated.By analyzing the variation of distance and contact time between ions in an electrolyte solution,from both spatial and temporal perspectives,new parameters were proposed to describe the interaction between ions:interaction distance(ID),and interaction time ratio(ITR).The best correlation between characteristic time ratio and solubility was found for a molar ratio of salt-to-water of 10:100 with a correlation coefficient of 0.96.For the same salt,a positive correlation was found between CTR and the molar ratio of salt and water.For type 1-1,type 2-1,type 1-2,and type 2-2 salts,the correlation coefficients between CTR and solubility were 0.93,0.96,0.92,and 0.98 for a salt-to-water molar ratio of 10:100,respectively.The solubility of multiple salts was predicted by simulations and compared with experimental values,yielding an average relative deviation of 12.4%.The new ion-interaction parameters offer significant advantages in describing strongly correlated and strongly hydrated electrolyte solutions.

Molecular Dynamics Simulation of Shock Response of CL-20 Co-crystals Containing Void Defects

To investigate the effect of void defects on the shock response of hexanitrohexaazaisowurtzitane(CL-20)co-crystals,shock responses of CL-20 co-crystals with energetic materials ligands trinitrotoluene(TNT),1,3-dinitrobenzene(DNB),solvents ligands dimethyl carbonate(DMC) and gamma-butyrolactone(GBL)with void were simulated,using molecular dynamics method and reactive force field.It is found that the CL-20 co-crystals with void defects will form hot spots when impacted,significantly affecting the decomposition of molecules around the void.The degree of molecular fragmentation is relatively low under the reflection velocity of 2 km/s,and the main reactions are the formation of dimer and the shedding of nitro groups.The existence of voids reduces the safety of CL-20 co-crystals,which induced the sensitivity of energetic co-crystals CL-20/TNT and CL-20/DNB to increase more significantly.Detonation has occurred under the reflection velocity of 4 km/s,energetic co-crystals are easier to polymerize than solvent co-crystals,and are not obviously affected by voids.The results show that the energy of the wave decreases after sweeping over the void,which reduces the chemical reaction frequency downstream of the void and affects the detonation performance,especially the solvent co-crystals.

Molecular Dynamics Numerical Simulation of Adsorption Characteristics and Exploitation Limits in Shale Oil Microscopic Pore Spaces

Microscopic pore structure in continental shale oil reservoirs is characterized by small pore throats and complex micro-structures.The adsorption behavior of hydrocarbons on the pore walls exhibits unique physical and chemical properties.Therefore,studying the adsorption morphology of hydrocarbon components in nanometer-sized pores and clarifying the exploitation limits of shale oil at the microscopic level are of great practical significance for the efficient development of continental shale oil.In this study,molecular dynamics simulations were employed to investigate the adsorption characteristics of various single-component shale oils in inorganic quartz fissures,and the influence of pore size and shale oil hydrocarbon composition on the adsorption properties in the pores was analyzed.The results show that different molecules have different adsorption capacities in shale oil pores,with lighter hydrocarbon components(C6H14)exhibiting stronger adsorption abilities.For the same adsorbed molecule,the adsorption amount linearly increases with the increase in pore diameter,but larger pores contribute more to shale oil adsorption.In shale pores,the thickness of the adsorption layer formed by shale oil molecules ranges from 0.4 to 0.5 nm,which is similar to the width of alkane molecules.Shale oil in the adsorbed state that is difficult to be exploited is mainly concentrated in the first adsorption layer.Among them,the volume fraction of adsorbed shale oil in 6 nm shale pores is 40.8%,while the volume fraction of shale oil that is difficult to be exploited is 16.2%.

Adaptive Optimal Discrete-Time Output-Feedback Using an Internal Model Principle and Adaptive Dynamic Programming

In order to address the output feedback issue for linear discrete-time systems, this work suggests a brand-new adaptive dynamic programming(ADP) technique based on the internal model principle(IMP). The proposed method, termed as IMP-ADP, does not require complete state feedback-merely the measurement of input and output data. More specifically, based on the IMP, the output control problem can first be converted into a stabilization problem. We then design an observer to reproduce the full state of the system by measuring the inputs and outputs. Moreover, this technique includes both a policy iteration algorithm and a value iteration algorithm to determine the optimal feedback gain without using a dynamic system model. It is important that with this concept one does not need to solve the regulator equation. Finally, this control method was tested on an inverter system of grid-connected LCLs to demonstrate that the proposed method provides the desired performance in terms of both tracking and disturbance rejection.

Exploring the molecular mechanism of action of curcumin for the treatment of diabetic retinopathy,using network pharmacology,molecular docking,and molecular dynamics simulation

Background:Based on network pharmacology and molecular docking,the present study investigated the mechanism of curcumin(CUR)in diabetic retinopathy treatment.Methods:Based on the DisGeNET,Swiss TargetPrediction,GeneCards,Online Mendelian Inheritance in Man,Gene Expression Omnibus,and Comparative Toxicogenomics Database,the intersection core targets of CUR and diabetic retinopathy were identified.The intersection target was imported into the STRING database to obtain the protein-protein interaction map.According to the Database for Annotation,Visualization and Integrated Discovery database,the intersected targets were enriched in Gene Ontology(GO)and Kyoto Encyclopedia of Genes and Genomes pathways.Then Cytoscape 3.9.1 is used to make the drug-target-disease-pathway network.The mechanism of CUR and diabetic retinopathy was further verified by molecular docking and molecular dynamics simulation.Results:There were 203 intersecting targets of CUR and diabetic retinopathy identified.1320 GO entries were enriched for GO functions,which were primarily involved in the composition of cells such as identical protein binding,protein binding,enzyme binding,etc.It was found that 175 pathways were enriched using Kyoto Encyclopedia of Genes and Genomes pathway enrichment methods,which were mainly included in the lipid and atherosclerosis,AGE-RAGE signaling pathway in diabetic complications,pathways in cancer,etc.In the molecular docking analysis,CUR was found to have a good ability to bind to the core targets of albumin,IL-1B,and IL-6.The binding of albumin to CUR was further verified by molecular dynamics simulation.Conclusion:As a result of this study,CUR may exert a role in the treatment of diabetic retinopathy through multi-target and multi-pathway regulation,which indicates a possible direction of future research.

Refinement of Adaptive Dynamical Simulation of Quantum Mechanical Double Slit Interference Phenomenon

We applied adaptive dynamics to double slit interference phenomenon using particle model and obtained partial successful results in our previous report. The patterns qualitatively corresponded well with experiments. Several properties such as concave single slit pattern and large influence of slight displacement of the emission position were different from the experimental results. In this study we tried other slit conditions and obtained consistent patterns with experiments. We do not claim that the adaptive dynamics is the principle of quantum mechanics, but the present results support the probability of adaptive dynamics as the candidate of the basis of quantum mechanics. We discuss the advantages of the adaptive dynamical view for foundations of quantum mechanics.

Exploring the moderating role of financial development in environmental Kuznets curve for South Africa:fresh evidence from the novel dynamic ARDL simulations approach

The extant literature has produced mixed evidence on the relationship between finan-cial development and ecological sustainability.This work addresses this conundrum by investigating financial development’s direct and indirect consequences on ecologi-cal quality utilizing the environmental Kuznets curve(EKC)methodological approach.Our empirical analysis is based on the novel dynamic autoregressive distributed lag simulations approach for South Africa between 1960 and 2020.The results,which used five distinct financial development measures,demonstrate that financial develop-ment boosts ecological integrity and environmental sustainability over the long and short terms.In the instance of South Africa,we additionally confirm the validity of the EKC theory.More importantly,the outcomes of the indirect channels demonstrate that financial development increases energy usage’s role in causing pollution while attenuating the detrimental impacts of economic growth,trade openness,and foreign direct investment on ecological quality.Moreover,the presence of an inadequate financial system is a requirement for the basis of the pollution haven hypothesis(PHH),which we examine using trade openness and foreign direct investment variables.PHH for both of these variables disappears when financial development crosses specified thresholds.Finally,industrial value addition destroys ecological quality while tech-nological innovation enhances it.This research provides some crucial policy recom-mendations and fresh perspectives for South Africa as it develops national initiatives to support ecological sustainability and reach its net zero emissions goal.

A stable implicit nodal integration-based particle finite element method(N-PFEM)for modelling saturated soil dynamics

In this study,we present a novel nodal integration-based particle finite element method(N-PFEM)designed for the dynamic analysis of saturated soils.Our approach incorporates the nodal integration technique into a generalised Hellinger-Reissner(HR)variational principle,creating an implicit PFEM formulation.To mitigate the volumetric locking issue in low-order elements,we employ a node-based strain smoothing technique.By discretising field variables at the centre of smoothing cells,we achieve nodal integration over cells,eliminating the need for sophisticated mapping operations after re-meshing in the PFEM.We express the discretised governing equations as a min-max optimisation problem,which is further reformulated as a standard second-order cone programming(SOCP)problem.Stresses,pore water pressure,and displacements are simultaneously determined using the advanced primal-dual interior point method.Consequently,our numerical model offers improved accuracy for stresses and pore water pressure compared to the displacement-based PFEM formulation.Numerical experiments demonstrate that the N-PFEM efficiently captures both transient and long-term hydro-mechanical behaviour of saturated soils with high accuracy,obviating the need for stabilisation or regularisation techniques commonly employed in other nodal integration-based PFEM approaches.This work holds significant implications for the development of robust and accurate numerical tools for studying saturated soil dynamics.

In situ TEM observations and molecular dynamics simulations of deformation defect activities in Mg via nanoindentation

In this work, we performed in situ nanoindentation in TEM to capture the real-time dislocation and twinning activities in pure Mg during loading and unloading. We demonstrated that the screw component of dislocations glides continuously, while the edge components rapidly become sessile during loading. The twin tip propagation is intermittent, whereas the twin boundary migration is more continuous. During unloading, we observed the elastic strain relaxation causes both dislocation retraction and detwinning. Moreover,we note that the plastic zone comprised of dislocations in Mg is well-defined, which contrasts with the diffused plastic zones observed in face-centered cubic metals under the nanoindentation impressions. Additionally, molecular dynamics simulations were performed to study the formation and evolution of deformation-induced crystallographic defects at the early stages of indentation. We observed that,in addition to dislocations, the I1stacking fault bounded with a <1/2c+p> Frank loop can be generated from the plastic zone ahead of the indenter, and potentially serve as a nucleation source for abundant dislocations observed experimentally. These new findings are anticipated to provide new knowledge on the deformation mechanisms of Mg, which are difficult to obtain through conventional ex situ approaches. These observations may serve as a baseline for simulation work that investigate the dynamics of dislocation slip and twinning in Mg and alloys.

Understanding the interfacial behaviors of benzene alkylation with butene using chloroaluminate ionic liquid catalyst: A molecular dynamics simulation

To better understand the benzene alkylation with chloroaluminate ionic liquids(ILs) as catalyst, the interfacial properties between the benzene/butene binary reactants and chloroaluminate ILs with varying cation alkyl chain length and different anions were investigated using molecular dynamics(MD) simulations. The results indicate that ILs can obviously improve the interfacial width, solubility and diffusion of reactants compared to H_(2)SO_(4). The longer alkyl chains of cations present a density enrichment at the interface and protrude into the binary reactants phase. Furthermore, the ILs consisting of 1-octyl-3-methylimidazolium cations([Omim]^(+)) and the stronger acidity heptachlorodialuminate anions([Al_(2)Cl_(7)]^(-)) are more beneficial to promote the interfacial width and facilitate the dissolution and diffusion of benzene in both the IL bulk and the interfacial region in comparison to the ones with shorter alkyl chains cations and weaker acidity anions. The information gives us a better guideline for the design of ILs for benzene alkylation.

Dynamic phasor-based hybrid simulation for multi-inverter grid-connected system

To realize the efficient transient simulation of a grid-connected power generation system based on multiple inverters, this paper proposes a hybrid simulation method integrating the models of electromagnetic transient and dynamic phasors. Based on a demonstration of the concepts and properties of dynamic phasors, the models of single-phase and three-phase inverters described by dynamic phasors are established first. Considering the numerical compatibility problem between dynamic phasors and instantaneous values, an interface scheme between dynamic phasors and instantaneous values is designed, and the efficiency and precision differences of various transformation methods are compared in detail.Finally, by utilizing MATLAB/Simulink, a hybrid simulation platform of a multi-inverter grid-connected system is built, and the efficiency and accuracy of the hybrid simulation are validated via comparison with the full electromagnetic transient simulation.

Molecular Dynamics Simulation of Interface Properties between Water-Based Inorganic Zinc Silicate Coating Modified by Organosilicone and Iron Substrate

The interface properties of Fe(101)/zinc silicate modified by organo-siloxane(KH-570)was studied by using the method of molecular dynamics simulation.By calculating the temperature and energy fluctuation of equilibrium state,equilibrium concentration distribution,MSD of layer and different groups,and interaction energy of two interface models,the influencing mechanism on the interface properties of adding organosiloxane into coating system was studied at the atomic scale.It shows that the temperature and energy of interface oscillated in a small range and it was exited in a state of dynamic equilibrium within the initial simulation stage(t<20 ps).It can be seen from the multiple peak states of concentration distribution that the iron substrate,organo-siloxane and zinc silicate are distributed in the form of a concentration gradient in the real environment.The rapid diffusion of free zinc powder in zinc silicate coating was the essential reason that affected the comprehensive properties of coating.The interface thickness decreased from 7.45 to 6.82Å,the MSD of free zinc powder was effectively reduced,and the interfacial energy was increased from 104.667 to 347.158 kcal/mol after being modified by organo-siloxane.

Molecular Dynamics Simulation of Hydrate Decomposition Under Action of Alcohol Inhibitors

The molecular dynamics method is used to investigate decomposition of methane hydrate at different temperatures,pressures and concentrations of inhibitor.By analyzing the parameters of system conformation,mean square displacement and radial distribution function,the decomposition of hydrate in the presence of alcohol inhibitors ethylene glycol and glycerol is explored.The results show that the hydroxyl groups in alcohol molecules can destroy the cage structure of hydrate,and form hydrogen bonds with nearby water molecules to effectively prevent the reformation of hydrate.Therefore,ethylene glycol and glycerol serve as inhibitors of methane hydrate,furthermore,in terms of inhibition effect,glycerol is better than ethylene glycol by comparing rate of hydrate decomposition.

Dynamic Mechanical Behavior and Numerical Simulation of an Ancient Underground Rock Mass under Impact Loading

To study the dynamic mechanical properties of tuff under different environmental conditions,the tuff from an ancient quarry in Shepan Island was prepared.The impact damage to the rock was tested using a triaxial dynamic impact mechanical testing system(TDIMTS)with different ground stresses,temperatures,and groundwater pressures.The time-strain relationship,dynamic stress-strain relationship,energy dissipation law,energy-peak strain relationship,and the impact damage pattern of the tuff specimens under impact air pressures were investigated.The TDIMTS experiment on ancient underground rock mass under impact loading was also simulated using the finite element analysis software LS-DYNA based on the Holmquist-Johnson-Cook(HJC)material model.The dynamic failure process,failure pattern and peak stress of tuff specimen were calculated.The simulation results obtained using the above methods were in good agreement with the experimental results.The results of the dynamic experiment show that with the same local stress,groundwater pressure,and temperature,the damage to the tuff specimens caused by blasting and quarrying disturbances gradually increases as the impact pressure increases.Under the same local stress,groundwater pressure,and temperature,the energy required to rupture the tuffs in ancient underground caverns is relatively small if the impact pressure is low accordingly,but as the impact pressure increases,the damage to the tuff caused by quarrying disturbance gradually increases.The damage gradually increases and the degree of damage to the tuff and the strain energy exhibit asymptotic growth when the tuff specimens are subjected to the greater strain energy,increasing the degree of rupturing of the tuff.In addition,the average crushing size decreases with increasing strain energy.By comparing the simulation results with the experimental results,it was found that the HJC model reflected the dynamic impact performance of tuff specimen,and the simulation results showed an evident strain rate effect.These results of this study can offer some guidance and theoretical support for the stability evaluation,protection,and safe operation of the ancient underground caverns in future.

Simulation-based Graphic Method for Dynamic System Modeling

The first step for engineering students to understand the practical system is to describe the system,which means to establish the mathematic model for the considered system.However,traditional mathematical modeling techniques are not readily assimilated by students,because the traditional modeling needs to obey the complicated physical laws for different engineering areas and requires a highly iterative process.In this paper,the simulation-based graphic modeling method is introduced to help students model the system and improve their performance in engineering education.The results from the questionnaire indicate that the simulation-based graphic method can help students to do the modeling of the system and improve their performance compared to the traditional ways.

Leveraging on Flock Dynamics and Farmers’ Production Constraints in Designing Flock Health Program for Accelerated Goat Production

Compliance with appropriate flock health program is vital for preventing introduction and minimizing impact of diseases in goat farms. Unfortunately, most goat farms in Uganda, especially in the Albertine Graben Zone lack flock health program. The associated frequent outbreaks slow down effort aimed at commercializing goat production. In this study, we documented flock dynamics, identified and prioritized pressing challenges experienced by goat farms during the year 2022 and generated appropriate flock health program and packaged it for dissemination to farmers. Materials and Methods: Using a cross-sectional design, semi-structured questionnaire, data were collected and analyzed with MS Excel 2013. The data included: location and socio-demographics of household, farming system, flock dynamics, housing, feeding system, health management, challenges encountered by goat farm and suggested solutions. Results: Beginning January 2022 to December 2022, the number of goats reared in all the 45 sample farms increased from 2128 to 2220 goats. Results showed that 884 kids were produced and 88 breeding goats were introduced into the farms. Three hundred ninety-nine goats died due to mainly diseases and 435 goats got withdrawn through nondeath. The average farm level and overall mortality rate were 21 goats per 1000 goatmonths and 15 goats per 1000 goatmonths respectively. The most pressing challenges encountered by sample farms were death of goats especially due to diseases, poor access to veterinary extension services, high cost of inputs and feed scarcity. Solutions suggested by sample farms were improved access to veterinary services, improved housing, enhanced vaccination of goats against diseases, and enhanced grazing land management and feed conservation, all of which were incorporated into the flock health program. Conclusion and Recommendations: An appropriate flock health program was generated based on flock dynamics and production constraints which reveal high mortality and limited access to veterinary services respectively. Actors are recommended to promote adoption and adherence to the flock health program so as to increase goat production and access to wider market.