Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

An Overview of Finite/Fixed-Time Control and Its Application in Engineering Systems

The finite/fixed-time stabilization and tracking control is currently a hot field in various systems since the faster convergence can be obtained. By contrast to the asymptotic stability,the finite-time stability possesses the better control performance and disturbance rejection property. Different from the finite-time stability, the fixed-time stability has a faster convergence speed and the upper bound of the settling time can be estimated. Moreover, the convergent time does not rely on the initial information.This work aims at presenting an overview of the finite/fixed-time stabilization and tracking control and its applications in engineering systems. Firstly, several fundamental definitions on the finite/fixed-time stability are recalled. Then, the research results on the finite/fixed-time stabilization and tracking control are reviewed in detail and categorized via diverse input signal structures and engineering applications. Finally, some challenging problems needed to be solved are presented.

Novel Parameter Identification Method for Basis Weight Control Loop of Papermaking Process

The basis weight control loop of the papermaking process is a non-linear system with time-delay and time-varying.It is impractical to identify a model that can restore the model of real papermaking process.Determining a more accurate identification model is very important for designing the controller of the control system and maintaining the stable operation of the papermaking process.In this study,a strange nonchaotic particle swarm optimization(SNPSO)algorithm is proposed to identify the models of real papermaking processes,and this identification ability is significantly enhanced compared with particle swarm optimization(PSO).First,random particles are initialized by strange nonchaotic sequences to obtain high-quality solutions.Furthermore,the weight of linear attenuation is replaced by strange nonchaotic sequence and the time-varying acceleration coefficients and a mutation rule with strange nonchaotic characteristics are utilized in SNPSO.The above strategies effectively improve the global and local search ability of particles and the ability to escape from local optimization.To illustrate the effectiveness of SNPSO,step response data are used to identify the models of real industrial processes.Compared with classical PSO,PSO with timevarying acceleration coefficients(PSO-TVAC)and modified particle swarm optimization(MPSO),the simulation results demonstrate that SNPSO has stronger identification ability,faster convergence speed,and better robustness.

Input Structure Design for Structural Controllability of Complex Networks

This paper addresses the problem of the input design of large-scale complex networks.Two types of network components,redundant inaccessible strongly connected component(RISCC)and intermittent inaccessible strongly connected component(IISCC)are defined,and a subnetwork called a driver network is developed.Based on these,an efficient method is proposed to find the minimum number of controlled nodes to achieve structural complete controllability of a network,in the case that each input can act on multiple state nodes.The range of the number of input nodes to achieve minimal control,and the configuration method(the connection between the input nodes and the controlled nodes)are presented.All possible input solutions can be obtained by this method.Moreover,we give an example and some experiments on real-world networks to illustrate the effectiveness of the method.

Constrained Moving Path Following Control for UAV With Robust Control Barrier Function

This paper studies the moving path following(MPF)problem for fixed-wing unmanned aerial vehicle(UAV)under output constraints and wind disturbances.The vehicle is required to converge to a reference path moving with respect to the inertial frame,while the path following error is not expected to violate the predefined boundaries.Differently from existing moving path following guidance laws,the proposed method removes complex geometric transformation by formulating the moving path following problem into a second-order time-varying control problem.A nominal moving path following guidance law is designed with disturbances and their derivatives estimated by high-order disturbance observers.To guarantee that the path following error will not exceed the prescribed bounds,a robust control barrier function is developed and incorporated into controller design with quadratic program based framework.The proposed method does not require the initial position of the UAV to be within predefined boundaries.And the safety margin concept makes error-constraint be respected even if in a noisy environment.The proposed guidance law is validated through numerical simulations of shipboard landing and hardware-in-theloop(HIL)experiments.

Neuro-Based Higher Order Sliding Mode Control for Perturbed Nonlinear Systems

One of the great concerns when tackling nonlinear systems is how to design a robust controller that is able to deal with uncertainty.Many researchers have been working on developing such type of controllers.One of the most effi-cient techniques employed to develop such controllers is sliding mode control(SMC).However,the low order SMC suffers from chattering problem which harm the actuators of the control system and thus unsuitable to be used in many practical applications.In this paper,the drawbacks of low order traditional sliding mode control(FOTSMC)are resolved by presenting a novel adaptive radial basis function neural network–based generalized rth order sliding mode control strategy for nth order uncertain nonlinear systems.The proposed solution adopts neural networks for their excellent capability in function approximation and thus used to approximate the nonlinearities and uncertainties for systems under considera-tion.The approximation errors are completely considered in the developed approach.The proposed approach can be used with any order of sliding mode and thus can be generally used with various types of applications.The global sta-bility of the proposed control approach is proved through Lyapunov stability cri-terion.The proposed approach is validated and assessed through simulations on the nonlinear inverted pendulum system with severe modeling uncertainties.The simulations results show that the proposed approach provide superior perfor-mance compared with other approaches in the literature.

A Fractional Order Fast Repetitive Control Paradigm of Vienna Rectifier for Power Quality Improvement

Due to attractive features,including high efficiency,low device stress,and ability to boost voltage,a Vienna rectifier is commonly employed as a battery charger in an electric vehicle(EV).However,the 6k±1 harmonics in the acside current of the Vienna rectifier deteriorate theTHDof the ac current,thus lowering the power factor.Therefore,the current closed-loop for suppressing 6k±1 harmonics is essential tomeet the desired total harmonic distortion(THD).Fast repetitive control(FRC)is generally adopted;however,the deviation of power grid frequency causes delay link in the six frequency fast repetitive control to become non-integer and the tracking performance to deteriorate.This paper presents the detailed parameter design and calculation of fractional order fast repetitive controller(FOFRC)for the non-integer delay link.The finite polynomial approximates the non-integer delay link through the Lagrange interpolation method.By comparing the frequency characteristics of traditional repetitive control,the effectiveness of the FOFRC strategy is verified.Finally,simulation and experiment validate the steadystate performance and harmonics suppression ability of FOFRC.

Energy Efficient and Intelligent Mosquito Repellent Fuzzy Control System

Mosquitoes are of great concern for occasionally carrying noxious diseases(dengue,malaria,zika,and yellow fever).To control mosquitoes,it is very crucial to effectively monitor their behavioral trends and presence.Traditional mosquito repellent works by heating small pads soaked in repellant,which then diffuses a protected area around you,a great alternative to spraying yourself with insecticide.But they have limitations,including the range,turning them on manually,and then waiting for the protection to kick in when the mosquitoes may find you.This research aims to design a fuzzy-based controller to solve the above issues by automatically determining a mosquito repellent’s speed and active time.The speed and active time depend on the repellent cartridge and the number of mosquitoes.The Mamdani model is used in the proposed fuzzy system(FS).The FS consists of identifying unambiguous inputs,a fuzzification process,rule evaluation,and a defuzzification process to produce unambiguous outputs.The input variables used are the repellent cartridge and the number of mosquitoes,and the speed of mosquito repellent is used as the output variable.The whole FS is designed and simulated using MATLAB Simulink R2016b.The proposed FS is executed and verified utilizing a microcontroller using its pulse width modulation capability.Different simulations of the proposed model are performed in many nonlinear processes.Then,a comparative analysis of the outcomes under similar conditions confirms the higher accuracy of the FS,yielding a maximum relative error of 10%.The experimental outcomes show that the root mean square error is reduced by 67.68%,and the mean absolute percentage error is reduced by 52.46%.Using a fuzzy-based mosquito repellent can help maintain the speed of mosquito repellent and control the energy used by the mosquito repellent.

Engineering Geometric Electrodes for Electric Field-Enhanced High-Performance Flexible In-Plane Micro-Supercapacitors

In plane micro-supercapacitors that are miniaturized energy storage components have attracted significant attention due to their high power densities for various ubiquitous and sustainable device systems as well as their facile integration on various flexible/wearable platform.To implement the micro-supercapacitors in various practical applications that can accompany solid state or gel electrolyte and flexible substrates,ions must be readily transported to electrodes for achieving high power densities.Herein,we show large enhancement in electrochemical properties of flexible,inplane micro-supercapacitor using sharp-edged interdigitated electrode design,which was simply fabricated through direct laser scribing method.The sharp-edged electrodes allowed strong electric field to be induced at the corners of the electrode fingers which led to the greater accumulation of ions near the surface of electrode,significantly enhancing the energy storage performance of micro-supercapacitors.The electric field-enhanced in-plane micro-supercapacitor showed the volumetric energy density of 1.52 Wh L^(−1)and the excellent cyclability with capacitive retention of 95.4%after 20000 cycles.We further showed various practicability of our sharp-edged design in micro-supercapacitors by showing circuit applicability,mechanical stability,and air stability.These results present an important pathway for designing electrodes in various energy storage devices.

Intelligent Networked Control of Vasoactive Drug Infusion for Patients with Uncertain Sensitivity

Abnormal high blood pressure or hypertension is still the leading risk factor for death and disability worldwide.This paper presents a new intelligent networked control of medical drug infusion system to regulate the mean arterial blood pressure for hypertensive patients with different health status conditions.The infusion of vasoactive drugs to patients endures various issues,such as variation of sensitivity and noise,which require effective and powerful systems to ensure robustness and good performance.The developed intelligent networked system is composed of a hybrid control scheme of interval type-2 fuzzy(IT2F)logic and teaching-learning-based optimization(TLBO)algorithm.This networked IT2F control is capable of managing the uncertain sensitivity of the patient to anti-hypertensive drugs successfully.To avoid the manual selection of control parameter values,the TLBO algorithm is mainly used to automatically find the best parameter values of the networked IT2F controller.The simulation results showed that the optimized networked IT2F achieved a good performance under external disturbances.A comparative study has also been conducted to emphasize the outperformance of the developed controller against traditional PID and type-1 fuzzy controllers.Moreover,the comparative evaluation demonstrated that the performance of the developed networked IT2F controller is superior to other control strategies in previous studies to handle unknown patients’sensitivity to infused vasoactive drugs in a noisy environment.

3D Path Optimisation of Unmanned Aerial Vehicles Using Q Learning-Controlled GWO-AOA

Unmanned Aerial Vehicles(UAVs)or drones introduced for military applications are gaining popularity in several other fields as well such as security and surveillance,due to their ability to perform repetitive and tedious tasks in hazardous environments.Their increased demand created the requirement for enabling the UAVs to traverse independently through the Three Dimensional(3D)flight environment consisting of various obstacles which have been efficiently addressed by metaheuristics in past literature.However,not a single optimization algorithms can solve all kind of optimization problem effectively.Therefore,there is dire need to integrate metaheuristic for general acceptability.To address this issue,in this paper,a novel reinforcement learning controlled Grey Wolf Optimisation-Archimedes Optimisation Algorithm(QGA)has been exhaustively introduced and exhaustively validated firstly on 22 benchmark functions and then,utilized to obtain the optimum flyable path without collision for UAVs in three dimensional environment.The performance of the developed QGA has been compared against the various metaheuristics.The simulation experimental results reveal that the QGA algorithm acquire a feasible and effective flyable path more efficiently in complicated environment.

Lightweight Malicious Code Classification Method Based on Improved Squeeze Net

With the growth of the Internet,more and more business is being done online,for example,online offices,online education and so on.While this makes people’s lives more convenient,it also increases the risk of the network being attacked by malicious code.Therefore,it is important to identify malicious codes on computer systems efficiently.However,most of the existing malicious code detection methods have two problems:(1)The ability of the model to extract features is weak,resulting in poor model performance.(2)The large scale of model data leads to difficulties deploying on devices with limited resources.Therefore,this paper proposes a lightweight malicious code identification model Lightweight Malicious Code Classification Method Based on Improved SqueezeNet(LCMISNet).In this paper,the MFire lightweight feature extraction module is constructed by proposing a feature slicing module and a multi-size depthwise separable convolution module.The feature slicing module reduces the number of parameters by grouping features.The multi-size depthwise separable convolution module reduces the number of parameters and enhances the feature extraction capability by replacing the standard convolution with depthwise separable convolution with different convolution kernel sizes.In addition,this paper also proposes a feature splicing module to connect the MFire lightweight feature extraction module based on the feature reuse and constructs the lightweight model LCMISNet.The malicious code recognition accuracy of LCMISNet on the BIG 2015 dataset and the Malimg dataset reaches 98.90% and 99.58%,respectively.It proves that LCMISNet has a powerful malicious code recognition performance.In addition,compared with other network models,LCMISNet has better performance,and a lower number of parameters and computations.

Design of a novel hybrid quantum deep neural network in INEQR images classification

We redesign the parameterized quantum circuit in the quantum deep neural network, construct a three-layer structure as the hidden layer, and then use classical optimization algorithms to train the parameterized quantum circuit, thereby propose a novel hybrid quantum deep neural network(HQDNN) used for image classification. After bilinear interpolation reduces the original image to a suitable size, an improved novel enhanced quantum representation(INEQR) is used to encode it into quantum states as the input of the HQDNN. Multi-layer parameterized quantum circuits are used as the main structure to implement feature extraction and classification. The output results of parameterized quantum circuits are converted into classical data through quantum measurements and then optimized on a classical computer. To verify the performance of the HQDNN, we conduct binary classification and three classification experiments on the MNIST(Modified National Institute of Standards and Technology) data set. In the first binary classification, the accuracy of 0 and 4 exceeds98%. Then we compare the performance of three classification with other algorithms, the results on two datasets show that the classification accuracy is higher than that of quantum deep neural network and general quantum convolutional neural network.

Decoding topological XYZ^(2) codes with reinforcement learning based on attention mechanisms

Quantum error correction, a technique that relies on the principle of redundancy to encode logical information into additional qubits to better protect the system from noise, is necessary to design a viable quantum computer. For this new topological stabilizer code-XYZ^(2) code defined on the cellular lattice, it is implemented on a hexagonal lattice of qubits and it encodes the logical qubits with the help of stabilizer measurements of weight six and weight two. However topological stabilizer codes in cellular lattice quantum systems suffer from the detrimental effects of noise due to interaction with the environment. Several decoding approaches have been proposed to address this problem. Here, we propose the use of a state-attention based reinforcement learning decoder to decode XYZ^(2) codes, which enables the decoder to more accurately focus on the information related to the current decoding position, and the error correction accuracy of our reinforcement learning decoder model under the optimisation conditions can reach 83.27% under the depolarizing noise model, and we have measured thresholds of 0.18856 and 0.19043 for XYZ^(2) codes at code spacing of 3–7 and 7–11, respectively. our study provides directions and ideas for applications of decoding schemes combining reinforcement learning attention mechanisms to other topological quantum error-correcting codes.

Quantum generative adversarial networks based on a readout error mitigation method with fault tolerant mechanism

Readout errors caused by measurement noise are a significant source of errors in quantum circuits,which severely affect the output results and are an urgent problem to be solved in noisy-intermediate scale quantum(NISQ)computing.In this paper,we use the bit-flip averaging(BFA)method to mitigate frequent readout errors in quantum generative adversarial networks(QGAN)for image generation,which simplifies the response matrix structure by averaging the qubits for each random bit-flip in advance,successfully solving problems with high cost of measurement for traditional error mitigation methods.Our experiments were simulated in Qiskit using the handwritten digit image recognition dataset under the BFA-based method,the Kullback-Leibler(KL)divergence of the generated images converges to 0.04,0.05,and 0.1 for readout error probabilities of p=0.01,p=0.05,and p=0.1,respectively.Additionally,by evaluating the fidelity of the quantum states representing the images,we observe average fidelity values of 0.97,0.96,and 0.95 for the three readout error probabilities,respectively.These results demonstrate the robustness of the model in mitigating readout errors and provide a highly fault tolerant mechanism for image generation models.

Recurrent neural network decoding of rotated surface codes based on distributed strategy

Quantum error correction is a crucial technology for realizing quantum computers.These computers achieve faulttolerant quantum computing by detecting and correcting errors using decoding algorithms.Quantum error correction using neural network-based machine learning methods is a promising approach that is adapted to physical systems without the need to build noise models.In this paper,we use a distributed decoding strategy,which effectively alleviates the problem of exponential growth of the training set required for neural networks as the code distance of quantum error-correcting codes increases.Our decoding algorithm is based on renormalization group decoding and recurrent neural network decoder.The recurrent neural network is trained through the ResNet architecture to improve its decoding accuracy.Then we test the decoding performance of our distributed strategy decoder,recurrent neural network decoder,and the classic minimum weight perfect matching(MWPM)decoder for rotated surface codes with different code distances under the circuit noise model,the thresholds of these three decoders are about 0.0052,0.0051,and 0.0049,respectively.Our results demonstrate that the distributed strategy decoder outperforms the other two decoders,achieving approximately a 5%improvement in decoding efficiency compared to the MWPM decoder and approximately a 2%improvement compared to the recurrent neural network decoder.

Prediction of collapse process and tipping points for mutualistic and competitive networks with k-core method

Ecosystems generally have the self-adapting ability to resist various external pressures or disturbances,which is always called resilience.However,once the external disturbances exceed the tipping points of the system resilience,the consequences would be catastrophic,and eventually lead the ecosystem to complete collapse.We capture the collapse process of ecosystems represented by plant-pollinator networks with the k-core nested structural method,and find that a sufficiently weak interaction strength or a sufficiently large competition weight can cause the structure of the ecosystem to collapse from its smallest k-core towards its largest k-core.Then we give the tipping points of structure and dynamic collapse of the entire system from the one-dimensional dynamic function of the ecosystem.Our work provides an intuitive and precise description of the dynamic process of ecosystem collapse under multiple interactions,and provides theoretical insights into further avoiding the occurrence of ecosystem collapse.

Survey and Prospect for Applying Knowledge Graph in Enterprise Risk Management

Enterprise risk management holds significant importance in fostering sustainable growth of businesses and in serving as a critical element for regulatory bodies to uphold market order.Amidst the challenges posed by intricate and unpredictable risk factors,knowledge graph technology is effectively driving risk management,leveraging its ability to associate and infer knowledge from diverse sources.This review aims to comprehensively summarize the construction techniques of enterprise risk knowledge graphs and their prominent applications across various business scenarios.Firstly,employing bibliometric methods,the aim is to uncover the developmental trends and current research hotspots within the domain of enterprise risk knowledge graphs.In the succeeding section,systematically delineate the technical methods for knowledge extraction and fusion in the standardized construction process of enterprise risk knowledge graphs.Objectively comparing and summarizing the strengths and weaknesses of each method,we provide recommendations for addressing the existing challenges in the construction process.Subsequently,categorizing the applied research of enterprise risk knowledge graphs based on research hotspots and risk category standards,and furnishing a detailed exposition on the applicability of technical routes and methods.Finally,the future research directions that still need to be explored in enterprise risk knowledge graphs were discussed,and relevant improvement suggestions were proposed.Practitioners and researchers can gain insights into the construction of technical theories and practical guidance of enterprise risk knowledge graphs based on this foundation.

Influencer identification of dynamical networks based on an information entropy dimension reduction method

Identifying critical nodes or sets in large-scale networks is a fundamental scientific problem and one of the key research directions in the fields of data mining and network science when implementing network attacks, defense, repair and control.Traditional methods usually begin from the centrality, node location or the impact on the largest connected component after node destruction, mainly based on the network structure.However, these algorithms do not consider network state changes.We applied a model that combines a random connectivity matrix and minimal low-dimensional structures to represent network connectivity.By using mean field theory and information entropy to calculate node activity,we calculated the overlap between the random parts and fixed low-dimensional parts to quantify the influence of node impact on network state changes and ranked them by importance.We applied this algorithm and the proposed importance algorithm to the overall analysis and stratified analysis of the C.elegans neural network.We observed a change in the critical entropy of the network state and by utilizing the proposed method we can calculate the nodes that indirectly affect muscle cells through neural layers.

Predicting Traffic Flow Using Dynamic Spatial-Temporal Graph Convolution Networks

Traffic flow prediction plays a key role in the construction of intelligent transportation system.However,due to its complex spatio-temporal dependence and its uncertainty,the research becomes very challenging.Most of the existing studies are based on graph neural networks that model traffic flow graphs and try to use fixed graph structure to deal with the relationship between nodes.However,due to the time-varying spatial correlation of the traffic network,there is no fixed node relationship,and these methods cannot effectively integrate the temporal and spatial features.This paper proposes a novel temporal-spatial dynamic graph convolutional network(TSADGCN).The dynamic time warping algorithm(DTW)is introduced to calculate the similarity of traffic flow sequence among network nodes in the time dimension,and the spatiotemporal graph of traffic flow is constructed to capture the spatiotemporal characteristics and dependencies of traffic flow.By combining graph attention network and time attention network,a spatiotemporal convolution block is constructed to capture spatiotemporal characteristics of traffic data.Experiments on open data sets PEMSD4 and PEMSD8 show that TSADGCN has higher prediction accuracy than well-known traffic flow prediction algorithms.