Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Reinforcement learning based adaptive control for uncertain mechanical systems with asymptotic tracking

This paper mainly focuses on the development of a learning-based controller for a class of uncertain mechanical systems modeled by the Euler-Lagrange formulation.The considered system can depict the behavior of a large class of engineering systems,such as vehicular systems,robot manipulators and satellites.All these systems are often characterized by highly nonlinear characteristics,heavy modeling uncertainties and unknown perturbations,therefore,accurate-model-based nonlinear control approaches become unavailable.Motivated by the challenge,a reinforcement learning(RL)adaptive control methodology based on the actor-critic framework is investigated to compensate the uncertain mechanical dynamics.The approximation inaccuracies caused by RL and the exogenous unknown disturbances are circumvented via a continuous robust integral of the sign of the error(RISE)control approach.Different from a classical RISE control law,a tanh(·)function is utilized instead of a sign(·)function to acquire a more smooth control signal.The developed controller requires very little prior knowledge of the dynamic model,is robust to unknown dynamics and exogenous disturbances,and can achieve asymptotic output tracking.Eventually,co-simulations through ADAMS and MATLAB/Simulink on a three degrees-of-freedom(3-DOF)manipulator and experiments on a real-time electromechanical servo system are performed to verify the performance of the proposed approach.

Toward Fine-grained Image Retrieval with Adaptive Deep Learning for Cultural Heritage Image

Fine-grained image classification is a challenging research topic because of the high degree of similarity among categories and the high degree of dissimilarity for a specific category caused by different poses and scales.A cul-tural heritage image is one of thefine-grained images because each image has the same similarity in most cases.Using the classification technique,distinguishing cultural heritage architecture may be difficult.This study proposes a cultural heri-tage content retrieval method using adaptive deep learning forfine-grained image retrieval.The key contribution of this research was the creation of a retrieval mod-el that could handle incremental streams of new categories while maintaining its past performance in old categories and not losing the old categorization of a cul-tural heritage image.The goal of the proposed method is to perform a retrieval task for classes.Incremental learning for new classes was conducted to reduce the re-training process.In this step,the original class is not necessary for re-train-ing which we call an adaptive deep learning technique.Cultural heritage in the case of Thai archaeological site architecture was retrieved through machine learn-ing and image processing.We analyze the experimental results of incremental learning forfine-grained images with images of Thai archaeological site architec-ture from world heritage provinces in Thailand,which have a similar architecture.Using afine-grained image retrieval technique for this group of cultural heritage images in a database can solve the problem of a high degree of similarity among categories and a high degree of dissimilarity for a specific category.The proposed method for retrieving the correct image from a database can deliver an average accuracy of 85 percent.Adaptive deep learning forfine-grained image retrieval was used to retrieve cultural heritage content,and it outperformed state-of-the-art methods infine-grained image retrieval.

XA-GANomaly: An Explainable Adaptive Semi-Supervised Learning Method for Intrusion Detection Using GANomaly

Intrusion detection involves identifying unauthorized network activity and recognizing whether the data constitute an abnormal network transmission.Recent research has focused on using semi-supervised learning mechanisms to identify abnormal network traffic to deal with labeled and unlabeled data in the industry.However,real-time training and classifying network traffic pose challenges,as they can lead to the degradation of the overall dataset and difficulties preventing attacks.Additionally,existing semi-supervised learning research might need to analyze the experimental results comprehensively.This paper proposes XA-GANomaly,a novel technique for explainable adaptive semi-supervised learning using GANomaly,an image anomalous detection model that dynamically trains small subsets to these issues.First,this research introduces a deep neural network(DNN)-based GANomaly for semi-supervised learning.Second,this paper presents the proposed adaptive algorithm for the DNN-based GANomaly,which is validated with four subsets of the adaptive dataset.Finally,this study demonstrates a monitoring system that incorporates three explainable techniques—Shapley additive explanations,reconstruction error visualization,and t-distributed stochastic neighbor embedding—to respond effectively to attacks on traffic data at each feature engineering stage,semi-supervised learning,and adaptive learning.Compared to other single-class classification techniques,the proposed DNN-based GANomaly achieves higher scores for Network Security Laboratory-Knowledge Discovery in Databases and UNSW-NB15 datasets at 13%and 8%of F1 scores and 4.17%and 11.51%for accuracy,respectively.Furthermore,experiments of the proposed adaptive learning reveal mostly improved results over the initial values.An analysis and monitoring system based on the combination of the three explainable methodologies is also described.Thus,the proposed method has the potential advantages to be applied in practical industry,and future research will explore handling unbalanced real-time datasets in various scenarios.

Adaptive Multi-Step Evaluation Design With Stability Guarantee for Discrete-Time Optimal Learning Control

This paper is concerned with a novel integrated multi-step heuristic dynamic programming(MsHDP)algorithm for solving optimal control problems.It is shown that,initialized by the zero cost function,MsHDP can converge to the optimal solution of the Hamilton-Jacobi-Bellman(HJB)equation.Then,the stability of the system is analyzed using control policies generated by MsHDP.Also,a general stability criterion is designed to determine the admissibility of the current control policy.That is,the criterion is applicable not only to traditional value iteration and policy iteration but also to MsHDP.Further,based on the convergence and the stability criterion,the integrated MsHDP algorithm using immature control policies is developed to accelerate learning efficiency greatly.Besides,actor-critic is utilized to implement the integrated MsHDP scheme,where neural networks are used to evaluate and improve the iterative policy as the parameter architecture.Finally,two simulation examples are given to demonstrate that the learning effectiveness of the integrated MsHDP scheme surpasses those of other fixed or integrated methods.

Adaptive Kernel Firefly Algorithm Based Feature Selection and Q-Learner Machine Learning Models in Cloud

CC’s(Cloud Computing)networks are distributed and dynamic as signals appear/disappear or lose significance.MLTs(Machine learning Techniques)train datasets which sometime are inadequate in terms of sample for inferring information.A dynamic strategy,DevMLOps(Development Machine Learning Operations)used in automatic selections and tunings of MLTs result in significant performance differences.But,the scheme has many disadvantages including continuity in training,more samples and training time in feature selections and increased classification execution times.RFEs(Recursive Feature Eliminations)are computationally very expensive in its operations as it traverses through each feature without considering correlations between them.This problem can be overcome by the use of Wrappers as they select better features by accounting for test and train datasets.The aim of this paper is to use DevQLMLOps for automated tuning and selections based on orchestrations and messaging between containers.The proposed AKFA(Adaptive Kernel Firefly Algorithm)is for selecting features for CNM(Cloud Network Monitoring)operations.AKFA methodology is demonstrated using CNSD(Cloud Network Security Dataset)with satisfactory results in the performance metrics like precision,recall,F-measure and accuracy used.

Scale adaptive fitness evaluation‐based particle swarm optimisation for hyperparameter and architecture optimisation in neural networks and deep learning

Research into automatically searching for an optimal neural network(NN)by optimi-sation algorithms is a significant research topic in deep learning and artificial intelligence.However,this is still challenging due to two issues:Both the hyperparameter and ar-chitecture should be optimised and the optimisation process is computationally expen-sive.To tackle these two issues,this paper focusses on solving the hyperparameter and architecture optimization problem for the NN and proposes a novel light‐weight scale‐adaptive fitness evaluation‐based particle swarm optimisation(SAFE‐PSO)approach.Firstly,the SAFE‐PSO algorithm considers the hyperparameters and architectures together in the optimisation problem and therefore can find their optimal combination for the globally best NN.Secondly,the computational cost can be reduced by using multi‐scale accuracy evaluation methods to evaluate candidates.Thirdly,a stagnation‐based switch strategy is proposed to adaptively switch different evaluation methods to better balance the search performance and computational cost.The SAFE‐PSO algorithm is tested on two widely used datasets:The 10‐category(i.e.,CIFAR10)and the 100−cate-gory(i.e.,CIFAR100).The experimental results show that SAFE‐PSO is very effective and efficient,which can not only find a promising NN automatically but also find a better NN than compared algorithms at the same computational cost.

A Study of Multimodal Intelligent Adaptive Learning System and Its Pattern of Promoting Learners’Online Learning Engagement

As the field of artificial intelligence continues to evolve,so too does the application of multimodal learning analysis and intelligent adaptive learning systems.This trend has the potential to promote the equalization of educational resources,the intellectualization of educational methods,and the modernization of educational reform,among other benefits.This study proposes a construction framework for an intelligent adaptive learning system that is supported by multimodal data.It provides a detailed explanation of the system’s working principles and patterns,which aim to enhance learners’online engagement in behavior,emotion,and cognition.The study seeks to address the issue of intelligent adaptive learning systems diagnosing learners’learning behavior based solely on learning achievement,to improve learners’online engagement,enable them to master more required knowledge,and ultimately achieve better learning outcomes.

Neurogenesis dynamics in the olfactory bulb:deciphering circuitry organization, function, and adaptive plasticity

Adult neurogenesis persists after birth in the subventricular zone, with new neurons migrating to the granule cell layer and glomerular layers of the olfactory bulb, where they integrate into existing circuitry as inhibitory interneurons. The generation of these new neurons in the olfactory bulb supports both structural and functional plasticity, aiding in circuit remodeling triggered by memory and learning processes. However, the presence of these neurons, coupled with the cellular diversity within the olfactory bulb, presents an ongoing challenge in understanding its network organization and function. Moreover,the continuous integration of new neurons in the olfactory bulb plays a pivotal role in regulating olfactory information processing. This adaptive process responds to changes in epithelial composition and contributes to the formation of olfactory memories by modulating cellular connectivity within the olfactory bulb and interacting intricately with higher-order brain regions. The role of adult neurogenesis in olfactory bulb functions remains a topic of debate. Nevertheless, the functionality of the olfactory bulb is intricately linked to the organization of granule cells around mitral and tufted cells. This organizational pattern significantly impacts output, network behavior, and synaptic plasticity, which are crucial for olfactory perception and memory. Additionally, this organization is further shaped by axon terminals originating from cortical and subcortical regions. Despite the crucial role of olfactory bulb in brain functions and behaviors related to olfaction, these complex and highly interconnected processes have not been comprehensively studied as a whole. Therefore, this manuscript aims to discuss our current understanding and explore how neural plasticity and olfactory neurogenesis contribute to enhancing the adaptability of the olfactory system. These mechanisms are thought to support olfactory learning and memory, potentially through increased complexity and restructuring of neural network structures, as well as the addition of new granule granule cells that aid in olfactory adaptation. Additionally, the manuscript underscores the importance of employing precise methodologies to elucidate the specific roles of adult neurogenesis amidst conflicting data and varying experimental paradigms. Understanding these processes is essential for gaining insights into the complexities of olfactory function and behavior.

Noise-Tolerant ZNN-Based Data-Driven Iterative Learning Control for Discrete Nonaffine Nonlinear MIMO Repetitive Systems

Aiming at the tracking problem of a class of discrete nonaffine nonlinear multi-input multi-output(MIMO) repetitive systems subjected to separable and nonseparable disturbances, a novel data-driven iterative learning control(ILC) scheme based on the zeroing neural networks(ZNNs) is proposed. First, the equivalent dynamic linearization data model is obtained by means of dynamic linearization technology, which exists theoretically in the iteration domain. Then, the iterative extended state observer(IESO) is developed to estimate the disturbance and the coupling between systems, and the decoupled dynamic linearization model is obtained for the purpose of controller synthesis. To solve the zero-seeking tracking problem with inherent tolerance of noise,an ILC based on noise-tolerant modified ZNN is proposed. The strict assumptions imposed on the initialization conditions of each iteration in the existing ILC methods can be absolutely removed with our method. In addition, theoretical analysis indicates that the modified ZNN can converge to the exact solution of the zero-seeking tracking problem. Finally, a generalized example and an application-oriented example are presented to verify the effectiveness and superiority of the proposed process.

Multiple Targets Localization Algorithm Based on Covariance Matrix Sparse Representation and Bayesian Learning

The multi-source passive localization problem is a problem of great interest in signal pro-cessing with many applications.In this paper,a sparse representation model based on covariance matrix is constructed for the long-range localization scenario,and a sparse Bayesian learning algo-rithm based on Laplace prior of signal covariance is developed for the base mismatch problem caused by target deviation from the initial point grid.An adaptive grid sparse Bayesian learning targets localization(AGSBL)algorithm is proposed.The AGSBL algorithm implements a covari-ance-based sparse signal reconstruction and grid adaptive localization dictionary learning.Simula-tion results show that the AGSBL algorithm outperforms the traditional compressed-aware localiza-tion algorithm for different signal-to-noise ratios and different number of targets in long-range scenes.

Adaptive Cloud Intrusion Detection System Based on Pruned Exact Linear Time Technique

Cloud computing environments,characterized by dynamic scaling,distributed architectures,and complex work-loads,are increasingly targeted by malicious actors.These threats encompass unauthorized access,data breaches,denial-of-service attacks,and evolving malware variants.Traditional security solutions often struggle with the dynamic nature of cloud environments,highlighting the need for robust Adaptive Cloud Intrusion Detection Systems(CIDS).Existing adaptive CIDS solutions,while offering improved detection capabilities,often face limitations such as reliance on approximations for change point detection,hindering their precision in identifying anomalies.This can lead to missed attacks or an abundance of false alarms,impacting overall security effectiveness.To address these challenges,we propose ACIDS(Adaptive Cloud Intrusion Detection System)-PELT.This novel Adaptive CIDS framework leverages the Pruned Exact Linear Time(PELT)algorithm and a Support Vector Machine(SVM)for enhanced accuracy and efficiency.ACIDS-PELT comprises four key components:(1)Feature Selection:Utilizing a hybrid harmony search algorithm and the symmetrical uncertainty filter(HSO-SU)to identify the most relevant features that effectively differentiate between normal and anomalous network traffic in the cloud environment.(2)Surveillance:Employing the PELT algorithm to detect change points within the network traffic data,enabling the identification of anomalies and potential security threats with improved precision compared to existing approaches.(3)Training Set:Labeled network traffic data forms the training set used to train the SVM classifier to distinguish between normal and anomalous behaviour patterns.(4)Testing Set:The testing set evaluates ACIDS-PELT’s performance by measuring its accuracy,precision,and recall in detecting security threats within the cloud environment.We evaluate the performance of ACIDS-PELT using the NSL-KDD benchmark dataset.The results demonstrate that ACIDS-PELT outperforms existing cloud intrusion detection techniques in terms of accuracy,precision,and recall.This superiority stems from ACIDS-PELT’s ability to overcome limitations associated with approximation and imprecision in change point detection while offering a more accurate and precise approach to detecting security threats in dynamic cloud environments.

Adaptive Optimal Output Regulation of Interconnected Singularly Perturbed Systems With Application to Power Systems

This article studies the adaptive optimal output regulation problem for a class of interconnected singularly perturbed systems(SPSs) with unknown dynamics based on reinforcement learning(RL).Taking into account the slow and fast characteristics among system states,the interconnected SPS is decomposed into the slow time-scale dynamics and the fast timescale dynamics through singular perturbation theory.For the fast time-scale dynamics with interconnections,we devise a decentralized optimal control strategy by selecting appropriate weight matrices in the cost function.For the slow time-scale dynamics with unknown system parameters,an off-policy RL algorithm with convergence guarantee is given to learn the optimal control strategy in terms of measurement data.By combining the slow and fast controllers,we establish the composite decentralized adaptive optimal output regulator,and rigorously analyze the stability and optimality of the closed-loop system.The proposed decomposition design not only bypasses the numerical stiffness but also alleviates the high-dimensionality.The efficacy of the proposed methodology is validated by a load-frequency control application of a two-area power system.

Tool Wear State Recognition with Deep Transfer Learning Based on Spindle Vibration for Milling Process

The wear of metal cutting tools will progressively rise as the cutting time goes on. Wearing heavily on the toolwill generate significant noise and vibration, negatively impacting the accuracy of the forming and the surfaceintegrity of the workpiece. Hence, during the cutting process, it is imperative to continually monitor the tool wearstate andpromptly replace anyheavilyworn tools toguarantee thequality of the cutting.The conventional tool wearmonitoring models, which are based on machine learning, are specifically built for the intended cutting conditions.However, these models require retraining when the cutting conditions undergo any changes. This method has noapplication value if the cutting conditions frequently change. This manuscript proposes a method for monitoringtool wear basedonunsuperviseddeep transfer learning. Due to the similarity of the tool wear process under varyingworking conditions, a tool wear recognitionmodel that can adapt to both current and previous working conditionshas been developed by utilizing cutting monitoring data from history. To extract and classify cutting vibrationsignals, the unsupervised deep transfer learning network comprises a one-dimensional (1D) convolutional neuralnetwork (CNN) with a multi-layer perceptron (MLP). To achieve distribution alignment of deep features throughthe maximum mean discrepancy algorithm, a domain adaptive layer is embedded in the penultimate layer of thenetwork. A platformformonitoring tool wear during endmilling has been constructed. The proposedmethod wasverified through the execution of a full life test of end milling under multiple working conditions with a Cr12MoVsteel workpiece. Our experiments demonstrate that the transfer learning model maintains a classification accuracyof over 80%. In comparisonwith the most advanced tool wearmonitoring methods, the presentedmodel guaranteessuperior performance in the target domains.

Artificial Intelligence-Enhanced Learning:A New Paradigm in the“Business Data Analysis and Application”Course

This paper explores the transformative impact of generative artificial intelligence(AI)on the“Business Data Analysis and Application”course in the post-2023 era,marking a significant paradigm shift in educational methodologies.It investigates how generative AI reshapes teaching and learning dynamics,enhancing the processing of complex data sets and nurturing critical thinking skills.The study highlights the role of AI in fostering dynamic,personalized,and adaptive learning experiences,addressing the evolving pedagogical needs of the business sector.Key challenges,including equitable access,academic integrity,and ethical considerations such as data privacy and algorithmic bias,are thoroughly examined.The research reveals that the integration of generative AI aligns with current professional demands,equipping students with cutting-edge AI tools,and tailoring learning to individual needs through real-time feedback mechanisms.The study concludes that the incorporation of generative AI into this course signifies a substantial evolution in educational approaches,offering profound implications for student learning and professional development.

Knowledge Transfer Learning via Dual Density Sampling for Resource-Limited Domain Adaptation

Most existing domain adaptation(DA) methods aim to explore favorable performance under complicated environments by sampling.However,there are three unsolved problems that limit their efficiencies:ⅰ) they adopt global sampling but neglect to exploit global and local sampling simultaneously;ⅱ)they either transfer knowledge from a global perspective or a local perspective,while overlooking transmission of confident knowledge from both perspectives;and ⅲ) they apply repeated sampling during iteration,which takes a lot of time.To address these problems,knowledge transfer learning via dual density sampling(KTL-DDS) is proposed in this study,which consists of three parts:ⅰ) Dual density sampling(DDS) that jointly leverages two sampling methods associated with different views,i.e.,global density sampling that extracts representative samples with the most common features and local density sampling that selects representative samples with critical boundary information;ⅱ)Consistent maximum mean discrepancy(CMMD) that reduces intra-and cross-domain risks and guarantees high consistency of knowledge by shortening the distances of every two subsets among the four subsets collected by DDS;and ⅲ) Knowledge dissemination(KD) that transmits confident and consistent knowledge from the representative target samples with global and local properties to the whole target domain by preserving the neighboring relationships of the target domain.Mathematical analyses show that DDS avoids repeated sampling during the iteration.With the above three actions,confident knowledge with both global and local properties is transferred,and the memory and running time are greatly reduced.In addition,a general framework named dual density sampling approximation(DDSA) is extended,which can be easily applied to other DA algorithms.Extensive experiments on five datasets in clean,label corruption(LC),feature missing(FM),and LC&FM environments demonstrate the encouraging performance of KTL-DDS.

改进Q-Learning输电线路超声驱鸟设备参数优化研究

超声波驱鸟是一种解决输电设备鸟害的重要手段,但现场使用超声波驱鸟器工作模式较单一,易产生鸟类适应问题。提出了一种改进Q-Learning输电线路超声驱鸟设备参数优化方法,针对涉鸟故障历史数据量少以及鸟类的适应性问题,将强化学习算法应用于输电线路超声驱鸟设备参数优化;针对传统强化学习算法在设备终端应用中存在收敛慢、耗时长的缺点,提出一种基于动态学习率的改进Q-Learning算法,对不同频段超声波的权重进行自适应优化。实验结果显示,改进Q-Learning算法最优参数的迭代收敛速度大幅提高,优化后驱鸟设备的驱鸟成功率达到了76%,优于传统强化学习算法模式,较好地解决了鸟类适应性问题。

Generalized projective synchronization of chaotic systems via adaptive learning control

In this paper, a learning control approach is applied to the generalized projective synchronisation （GPS） of different chaotic systems with unknown periodically time-varying parameters. Using the Lyapunov--Krasovskii functional stability theory, a differential-difference mixed parametric learning law and an adaptive learning control law are constructed to make the states of two different chaotic systems asymptotically synchronised. The scheme is successfully applied to the generalized projective synchronisation between the Lorenz system and Chen system. Moreover, numerical simulations results are used to verify the effectiveness of the proposed scheme.

Adaptive optics based on machine learning: a review

Adaptive optics techniques have been developed over the past half century and routinely used in large ground-based telescopes for more than 30 years.Although this technique has already been used in various applications,the basic setup and methods have not changed over the past 40 years.In recent years,with the rapid development of artificial in-telligence,adaptive optics will be boosted dramatically.In this paper,the recent advances on almost all aspects of adapt-ive optics based on machine learning are summarized.The state-of-the-art performance of intelligent adaptive optics are reviewed.The potential advantages and deficiencies of intelligent adaptive optics are also discussed.

Observer-based Adaptive Iterative Learning Control for Nonlinear Systems with Time-varying Delays

An observer-based adaptive iterative learning control （AILC） scheme is developed for a class of nonlinear systems with unknown time-varying parameters and unknown time-varying delays. The linear matrix inequality （LMI） method is employed to design the nonlinear observer. The designed controller contains a proportional-integral-derivative （PID） feedback term in time domain. The learning law of unknown constant parameter is differential-difference-type, and the learning law of unknown time-varying parameter is difference-type. It is assumed that the unknown delay-dependent uncertainty is nonlinearly parameterized. By constructing a Lyapunov-Krasovskii-like composite energy function （CEF）, we prove the boundedness of all closed-loop signals and the convergence of tracking error. A simulation example is provided to illustrate the effectiveness of the control algorithm proposed in this paper.