Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Adaptive Optimal Discrete-Time Output-Feedback Using an Internal Model Principle and Adaptive Dynamic Programming

In order to address the output feedback issue for linear discrete-time systems, this work suggests a brand-new adaptive dynamic programming(ADP) technique based on the internal model principle(IMP). The proposed method, termed as IMP-ADP, does not require complete state feedback-merely the measurement of input and output data. More specifically, based on the IMP, the output control problem can first be converted into a stabilization problem. We then design an observer to reproduce the full state of the system by measuring the inputs and outputs. Moreover, this technique includes both a policy iteration algorithm and a value iteration algorithm to determine the optimal feedback gain without using a dynamic system model. It is important that with this concept one does not need to solve the regulator equation. Finally, this control method was tested on an inverter system of grid-connected LCLs to demonstrate that the proposed method provides the desired performance in terms of both tracking and disturbance rejection.

Dynamic Location Method for Shallow Ocean Bottom Nodes Using the Levenberg-Marquart Algorithm

Ocean bottom node(OBN)data acquisition is the main development direction of marine seismic exploration;it is widely promoted,especially in shallow sea environments.However,the OBN receivers may move several times because they are easily affected by tides,currents,and other factors in the shallow sea environment during long-term acquisition.If uncorrected,then the imaging quality of subsequent processing will be affected.The conventional secondary positioning does not consider the case of multiple movements of the receivers,and the accuracy of secondary positioning is insufficient.The first arrival wave of OBN seismic data in shallow ocean mainly comprises refracted waves.In this study,a nonlinear model is established in accordance with the propagation mechanism of a refracted wave and its relationship with the time interval curve to realize the accurate location of multiple receiver movements.In addition,the Levenberg-Marquart algorithm is used to reduce the influence of the first arrival pickup error and to automatically detect the receiver movements,identifying the accurate dynamic relocation of the receivers.The simulation and field data show that the proposed method can realize the dynamic location of multiple receiver movements,thereby improving the accuracy of seismic imaging and achieving high practical value.

MADDPG-D2: An Intelligent Dynamic Task Allocation Algorithm Based on Multi-Agent Architecture Driven by Prior Knowledge

Aiming at the problems of low solution accuracy and high decision pressure when facing large-scale dynamic task allocation(DTA)and high-dimensional decision space with single agent,this paper combines the deep reinforce-ment learning(DRL)theory and an improved Multi-Agent Deep Deterministic Policy Gradient(MADDPG-D2)algorithm with a dual experience replay pool and a dual noise based on multi-agent architecture is proposed to improve the efficiency of DTA.The algorithm is based on the traditional Multi-Agent Deep Deterministic Policy Gradient(MADDPG)algorithm,and considers the introduction of a double noise mechanism to increase the action exploration space in the early stage of the algorithm,and the introduction of a double experience pool to improve the data utilization rate;at the same time,in order to accelerate the training speed and efficiency of the agents,and to solve the cold-start problem of the training,the a priori knowledge technology is applied to the training of the algorithm.Finally,the MADDPG-D2 algorithm is compared and analyzed based on the digital battlefield of ground and air confrontation.The experimental results show that the agents trained by the MADDPG-D2 algorithm have higher win rates and average rewards,can utilize the resources more reasonably,and better solve the problem of the traditional single agent algorithms facing the difficulty of solving the problem in the high-dimensional decision space.The MADDPG-D2 algorithm based on multi-agent architecture proposed in this paper has certain superiority and rationality in DTA.

Structure-preserving algorithms for guiding center dynamics based on the slow manifold of classical Pauli particle

The classical Pauli particle(CPP) serves as a slow manifold, substituting the conventional guiding center dynamics. Based on the CPP, we utilize the averaged vector field(AVF) method in the computations of drift orbits. Demonstrating significantly higher efficiency, this advanced method is capable of accomplishing the simulation in less than one-third of the time of directly computing the guiding center motion. In contrast to the CPP-based Boris algorithm, this approach inherits the advantages of the AVF method, yielding stable trajectories even achieved with a tenfold time step and reducing the energy error by two orders of magnitude. By comparing these two CPP algorithms with the traditional RK4 method, the numerical results indicate a remarkable performance in terms of both the computational efficiency and error elimination. Moreover, we verify the properties of slow manifold integrators and successfully observe the bounce on both sides of the limiting slow manifold with deliberately chosen perturbed initial conditions. To evaluate the practical value of the methods, we conduct simulations in non-axisymmetric perturbation magnetic fields as part of the experiments,demonstrating that our CPP-based AVF method can handle simulations under complex magnetic field configurations with high accuracy, which the CPP-based Boris algorithm lacks. Through numerical experiments, we demonstrate that the CPP can replace guiding center dynamics in using energy-preserving algorithms for computations, providing a new, efficient, as well as stable approach for applying structure-preserving algorithms in plasma simulations.

Flexible Job Shop Composite Dispatching Rule Mining Approach Based on an Improved Genetic Programming Algorithm

To obtain a suitable scheduling scheme in an effective time range,the minimum completion time is taken as the objective of Flexible Job Shop scheduling Problems(FJSP)with different scales,and Composite Dispatching Rules(CDRs)are applied to generate feasible solutions.Firstly,the binary tree coding method is adopted,and the constructed function set is normalized.Secondly,a CDR mining approach based on an Improved Genetic Programming Algorithm(IGPA)is designed.Two population initialization methods are introduced to enrich the initial population,and a superior and inferior population separation strategy is designed to improve the global search ability of the algorithm.At the same time,two individual mutation methods are introduced to improve the algorithm’s local search ability,to achieve the balance between global search and local search.In addition,the effectiveness of the IGPA and the superiority of CDRs are verified through comparative analysis.Finally,Deep Reinforcement Learning(DRL)is employed to solve the FJSP by incorporating the CDRs as the action set,the selection times are counted to further verify the superiority of CDRs.

Multi-Label Feature Selection Based on Improved Ant Colony Optimization Algorithm with Dynamic Redundancy and Label Dependence

The world produces vast quantities of high-dimensional multi-semantic data.However,extracting valuable information from such a large amount of high-dimensional and multi-label data is undoubtedly arduous and challenging.Feature selection aims to mitigate the adverse impacts of high dimensionality in multi-label data by eliminating redundant and irrelevant features.The ant colony optimization algorithm has demonstrated encouraging outcomes in multi-label feature selection,because of its simplicity,efficiency,and similarity to reinforcement learning.Nevertheless,existing methods do not consider crucial correlation information,such as dynamic redundancy and label correlation.To tackle these concerns,the paper proposes a multi-label feature selection technique based on ant colony optimization algorithm(MFACO),focusing on dynamic redundancy and label correlation.Initially,the dynamic redundancy is assessed between the selected feature subset and potential features.Meanwhile,the ant colony optimization algorithm extracts label correlation from the label set,which is then combined into the heuristic factor as label weights.Experimental results demonstrate that our proposed strategies can effectively enhance the optimal search ability of ant colony,outperforming the other algorithms involved in the paper.

Vehicle routing optimization algorithm based on time windows and dynamic demand

To provide the supplier with the minimizum vehicle travel distance in the distribution process of goods in three situations of new customer demand,customer cancellation service,and change of customer delivery address,based on the ideas of pre-optimization and real-time optimization,a two-stage planning model of dynamic demand based vehicle routing problem with time windows was established.At the pre-optimization stage,an improved genetic algorithm was used to obtain the pre-optimized distribution route,a large-scale neighborhood search method was integrated into the mutation operation to improve the local optimization performance of the genetic algorithm,and a variety of operators were introduced to expand the search space of neighborhood solutions;At the real-time optimization stage,a periodic optimization strategy was adopted to transform a complex dynamic problem into several static problems,and four neighborhood search operators were used to quickly adjust the route.Two different scale examples were designed for experiments.It is proved that the algorithm can plan the better route,and adjust the distribution route in time under the real-time constraints.Therefore,the proposed algorithm can provide theoretical guidance for suppliers to solve the dynamic demand based vehicle routing problem.

Enhanced asphalt dynamic modulus prediction: A detailed analysis of artificial hummingbird algorithm-optimised boosted trees

This study introduces and evaluates a novel artificial hummingbird algorithm-optimised boosted tree(AHAboosted)model for predicting the dynamic modulus(E*)of hot mix asphalt concrete.Using a substantial dataset from NCHRP Report-547,the model was trained and rigorously tested.Performance metrics,specifically RMSE,MAE,and R2,were employed to assess the model's predictive accuracy,robustness,and generalisability.When benchmarked against well-established models like support vector machines(SVM)and gaussian process regression(GPR),the AHA-boosted model demonstrated enhanced performance.It achieved R2 values of 0.997 in training and 0.974 in testing,using the traditional Witczak NCHRP 1-40D model inputs.Incorporating features such as test temperature,frequency,and asphalt content led to a 1.23%increase in the test R2,signifying an improvement in the model's accuracy.The study also explored feature importance and sensitivity through SHAP and permutation importance plots,highlighting binder complex modulus|G*|as a key predictor.Although the AHA-boosted model shows promise,a slight decrease in R2 from training to testing indicates a need for further validation.Overall,this study confirms the AHA-boosted model as a highly accurate and robust tool for predicting the dynamic modulus of hot mix asphalt concrete,making it a valuable asset for pavement engineering.

Systematic Review:Load Balancing in Cloud Computing by Using Metaheuristic Based Dynamic Algorithms

Cloud Computing has the ability to provide on-demand access to a shared resource pool.It has completely changed the way businesses are managed,implement applications,and provide services.The rise in popularity has led to a significant increase in the user demand for services.However,in cloud environments efficient load balancing is essential to ensure optimal performance and resource utilization.This systematic review targets a detailed description of load balancing techniques including static and dynamic load balancing algorithms.Specifically,metaheuristic-based dynamic load balancing algorithms are identified as the optimal solution in case of increased traffic.In a cloud-based context,this paper describes load balancing measurements,including the benefits and drawbacks associated with the selected load balancing techniques.It also summarizes the algorithms based on implementation,time complexity,adaptability,associated issue(s),and targeted QoS parameters.Additionally,the analysis evaluates the tools and instruments utilized in each investigated study.Moreover,comparative analysis among static,traditional dynamic and metaheuristic algorithms based on response time by using the CloudSim simulation tool is also performed.Finally,the key open problems and potential directions for the state-of-the-art metaheuristic-based approaches are also addressed.

Dynamic Pricing Strategies Using Artificial Intelligence Algorithm

Pricing strategies can have a huge impact on a company’s success. This paper focuses on the advantages and disadvantages of using artificial intelligence in dynamic pricing strategies. A good understanding of the possible benefits and challenges will help companies to understand the impact of their chosen pricing strategies. AI-driven Dynamic pricing has great opportunities to increase a firm’s profits. Firms can benefit from personalized pricing based on personal behavior and characteristics, as well as cost reduction by increasing efficiency and reducing the need to use manual work and automation. However, AI-driven dynamic rewarding can have a negative impact on customers’ perception of trust, fairness and transparency. Since price discrimination is used, ethical issues such as privacy and equity may arise. Understanding the businesses and customers that determine pricing strategy is so important that one cannot exist without the other. It will provide a comprehensive overview of the main advantages and disadvantages of AI-assisted dynamic pricing strategy. The main objective of this research is to uncover the most notable advantages and disadvantages of implementing AI-enabled dynamic pricing strategies. Future research can extend the understanding of algorithmic pricing through case studies. In this way, new, practical implications can be developed in the future. It is important to investigate how issues related to customers’ trust and feelings of unfairness can be mitigated, for example by price framing.

Shuffled frog leaping algorithm with non-dominated sorting for dynamic weapon-target assignment

The dynamic weapon target assignment(DWTA)problem is of great significance in modern air combat.However,DWTA is a highly complex constrained multi-objective combinatorial optimization problem.An improved elitist non-dominated sorting genetic algorithm-II(NSGA-II)called the non-dominated shuffled frog leaping algorithm(NSFLA)is proposed to maximize damage to enemy targets and minimize the self-threat in air combat constraints.In NSFLA,the shuffled frog leaping algorithm(SFLA)is introduced to NSGA-II to replace the inside evolutionary scheme of the genetic algorithm(GA),displaying low optimization speed and heterogeneous space search defects.Two improvements have also been raised to promote the internal optimization performance of SFLA.Firstly,the local evolution scheme,a novel crossover mechanism,ensures that each individual participates in updating instead of only the worst ones,which can expand the diversity of the population.Secondly,a discrete adaptive mutation algorithm based on the function change rate is applied to balance the global and local search.Finally,the scheme is verified in various air combat scenarios.The results show that the proposed NSFLA has apparent advantages in solution quality and efficiency,especially in many aircraft and the dynamic air combat environment.

UAV flight strategy algorithm based on dynamic programming

Unmanned aerial vehicles(UAVs) may play an important role in data collection and offloading in vast areas deploying wireless sensor networks, and the UAV’s action strategy has a vital influence on achieving applicability and computational complexity. Dynamic programming(DP) has a good application in the path planning of UAV, but there are problems in the applicability of special terrain environment and the complexity of the algorithm.Based on the analysis of DP, this paper proposes a hierarchical directional DP(DDP) algorithm based on direction determination and hierarchical model. We compare our methods with Q-learning and DP algorithm by experiments, and the results show that our method can improve the terrain applicability, meanwhile greatly reduce the computational complexity.

Primal-Dual Interior-Point Algorithms with Dynamic Step-Size Based on Kernel Functions for Linear Programming

In this paper, primal-dual interior-point algorithm with dynamic step size is implemented for linear programming （LP） problems. The algorithms are based on a few kernel functions, including both serf-regular functions and non-serf-regular ones. The dynamic step size is compared with fixed step size for the algorithms in inner iteration of Newton step. Numerical tests show that the algorithms with dynaraic step size are more efficient than those with fixed step size.

A stable implicit nodal integration-based particle finite element method(N-PFEM)for modelling saturated soil dynamics

In this study,we present a novel nodal integration-based particle finite element method(N-PFEM)designed for the dynamic analysis of saturated soils.Our approach incorporates the nodal integration technique into a generalised Hellinger-Reissner(HR)variational principle,creating an implicit PFEM formulation.To mitigate the volumetric locking issue in low-order elements,we employ a node-based strain smoothing technique.By discretising field variables at the centre of smoothing cells,we achieve nodal integration over cells,eliminating the need for sophisticated mapping operations after re-meshing in the PFEM.We express the discretised governing equations as a min-max optimisation problem,which is further reformulated as a standard second-order cone programming(SOCP)problem.Stresses,pore water pressure,and displacements are simultaneously determined using the advanced primal-dual interior point method.Consequently,our numerical model offers improved accuracy for stresses and pore water pressure compared to the displacement-based PFEM formulation.Numerical experiments demonstrate that the N-PFEM efficiently captures both transient and long-term hydro-mechanical behaviour of saturated soils with high accuracy,obviating the need for stabilisation or regularisation techniques commonly employed in other nodal integration-based PFEM approaches.This work holds significant implications for the development of robust and accurate numerical tools for studying saturated soil dynamics.

Novel algorithm for distributed replicas management based on dynamic programming

Replicas can improve the data reliability in distributed system. However, the traditional algorithms for replica management are based on the assumption that all replicas have the uniform reliability, which is inaccurate in some actual systems. To address such problem, a novel algorithm is proposed based on dynamic programming to manage the number and distribution of replicas in different nodes. By using Markov model, replicas management is organized as a multi-phase process, and the recursion equations are provided. In this algorithm, the heterogeneity of nodes, the expense for maintaining replicas and the engaged space have been considered. Under these restricted conditions, this algorithm realizes high data reliability in a distributed system. The results of case analysis prove the feasibility of the algorithm.

Application of Dynamic Programming Algorithm Based on Model Predictive Control in Hybrid Electric Vehicle Control Strategy

A good hybrid vehicle control strategy cannot only meet the power requirements of the vehicle,but also effectively save fuel and reduce emissions.In this paper,the construction of model predictive control in hybrid electric vehicle is proposed.The solving process and the use of reference trajectory are discussed for the application of MPC based on dynamic programming algorithm.The simulation of hybrid electric vehicle is carried out under a specific working condition.The simulation results show that the control strategy can effectively reduce fuel consumption when the torque of engine and motor is reasonably distributed,and the effectiveness of the control strategy is verified.

Optimal Dynamic Voltage Restorer Using Water Cycle Optimization Algorithm

This paper proposes a low complexity control scheme for voltage control of a dynamic voltage restorer(DVR)in a three-phase system.The control scheme employs the fractional order,proportional-integral-derivative(FOPID)controller to improve on the DVR performance in order to enhance the power quality in terms of the response time,steady-state error and total harmonic distortion(THD).The result obtained was compared with fractional order,proportionalintegral(FOPI),proportional-integral-derivative(PID)and proportional-integral(PI)controllers in order to show the effectiveness of the proposed DVR control scheme.A water cycle optimization algorithm(WCA)was utilized to find the optimal set for all the controller gains.They were used to solve four power quality issues;balanced voltage sag,balanced voltage swell,unbalanced voltage sag,and unbalanced voltage swell.It showed that one set of controller gain obtained from the WCA could solve all the power quality issues while the others in the literature needed an individual set of optimal gain for each power quality problem.To prove the concept,the proposed DVR algorithm was simulated in the MATLAB/Simulink software and the results revealed that the four optimal controllers can compensate for all the power quality problems.A comparative analysis of the results in various aspects of their dynamic response and%THD was discussed and analyzed.It was found that PID controller yields the most rapid performance in terms of average response time while FOPID controller yields the best performance in term of average%steady-state error.FOPI controller was found to provide the lowest THD percentage in the average%THD.FOPID did not differ much in average response from the PID and average%THD from FOPI;however,FOPID provided the most outstanding average steady-state error.According to the CBMA curve,the dynamic responses of all controllers fall in the acceptable power quality area.The total harmonic distortion(THD)of the compensated load voltage from all the controllers were within the 8%limit in accordance to the IEEE std.519-2014.

Hierarchical adaptive stereo matching algorithm for obstacle detection with dynamic programming

An adaptive weighted stereo matching algorithm with multilevel and bidirectional dynamic programming based on ground control points （GCPs） is presented. To decrease time complexity without losing matching precision, using a multilevel search scheme, the coarse matching is processed in typical disparity space image, while the fine matching is processed in disparity-offset space image. In the upper level, GCPs are obtained by enhanced volumetric iterative algorithm enforcing the mutual constraint and the threshold constraint. Under the supervision of the highly reliable GCPs, bidirectional dynamic programming framework is employed to solve the inconsistency in the optimization path. In the lower level, to reduce running time, disparity-offset space is proposed to efficiently achieve the dense disparity image. In addition, an adaptive dual support-weight strategy is presented to aggregate matching cost, which considers photometric and geometric information. Further, post-processing algorithm can ameliorate disparity results in areas with depth discontinuities and related by occlusions using dual threshold algorithm, where missing stereo information is substituted from surrounding regions. To demonstrate the effectiveness of the algorithm, we present the two groups of experimental results for four widely used standard stereo data sets, including discussion on performance and comparison with other methods, which show that the algorithm has not only a fast speed, but also significantly improves the efficiency of holistic optimization.

A dynamic fusion path planning algorithm for mobile robots incorporating improved IB-RRT∗and deep reinforcement learning

Dynamic path planning is crucial for mobile robots to navigate successfully in unstructured envi-ronments.To achieve globally optimal path and real-time dynamic obstacle avoidance during the movement,a dynamic path planning algorithm incorporating improved IB-RRT∗and deep reinforce-ment learning(DRL)is proposed.Firstly,an improved IB-RRT∗algorithm is proposed for global path planning by combining double elliptic subset sampling and probabilistic central circle target bi-as.Then,to tackle the slow response to dynamic obstacles and inadequate obstacle avoidance of tra-ditional local path planning algorithms,deep reinforcement learning is utilized to predict the move-ment trend of dynamic obstacles,leading to a dynamic fusion path planning.Finally,the simulation and experiment results demonstrate that the proposed improved IB-RRT∗algorithm has higher con-vergence speed and search efficiency compared with traditional Bi-RRT∗,Informed-RRT∗,and IB-RRT∗algorithms.Furthermore,the proposed fusion algorithm can effectively perform real-time obsta-cle avoidance and navigation tasks for mobile robots in unstructured environments.