Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Cooperative User-Scheduling and Resource Allocation Optimization for Intelligent Reflecting Surface Enhanced LEO Satellite Communication

Lower Earth Orbit(LEO) satellite becomes an important part of complementing terrestrial communication due to its lower orbital altitude and smaller propagation delay than Geostationary satellite. However, the LEO satellite communication system cannot meet the requirements of users when the satellite-terrestrial link is blocked by obstacles. To solve this problem, we introduce Intelligent reflect surface(IRS) for improving the achievable rate of terrestrial users in LEO satellite communication. We investigated joint IRS scheduling, user scheduling, power and bandwidth allocation(JIRPB) optimization algorithm for improving LEO satellite system throughput.The optimization problem of joint user scheduling and resource allocation is formulated as a non-convex optimization problem. To cope with this problem, the nonconvex optimization problem is divided into resource allocation optimization sub-problem and scheduling optimization sub-problem firstly. Second, we optimize the resource allocation sub-problem via alternating direction multiplier method(ADMM) and scheduling sub-problem via Lagrangian dual method repeatedly.Third, we prove that the proposed resource allocation algorithm based ADMM approaches sublinear convergence theoretically. Finally, we demonstrate that the proposed JIRPB optimization algorithm improves the LEO satellite communication system throughput.

Low-Cost Federated Broad Learning for Privacy-Preserved Knowledge Sharing in the RIS-Aided Internet of Vehicles

High-efficiency and low-cost knowledge sharing can improve the decision-making ability of autonomous vehicles by mining knowledge from the Internet of Vehicles(IoVs).However,it is challenging to ensure high efficiency of local data learning models while preventing privacy leakage in a high mobility environment.In order to protect data privacy and improve data learning efficiency in knowledge sharing,we propose an asynchronous federated broad learning(FBL)framework that integrates broad learning(BL)into federated learning(FL).In FBL,we design a broad fully connected model(BFCM)as a local model for training client data.To enhance the wireless channel quality for knowledge sharing and reduce the communication and computation cost of participating clients,we construct a joint resource allocation and reconfigurable intelligent surface(RIS)configuration optimization framework for FBL.The problem is decoupled into two convex subproblems.Aiming to improve the resource scheduling efficiency in FBL,a double Davidon–Fletcher–Powell(DDFP)algorithm is presented to solve the time slot allocation and RIS configuration problem.Based on the results of resource scheduling,we design a reward-allocation algorithm based on federated incentive learning(FIL)in FBL to compensate clients for their costs.The simulation results show that the proposed FBL framework achieves better performance than the comparison models in terms of efficiency,accuracy,and cost for knowledge sharing in the IoV.

Ensuring Secure Platooning of Constrained Intelligent and Connected Vehicles Against Byzantine Attacks:A Distributed MPC Framework

This study investigates resilient platoon control for constrained intelligent and connected vehicles(ICVs)against F-local Byzantine attacks.We introduce a resilient distributed model-predictive platooning control framework for such ICVs.This framework seamlessly integrates the predesigned optimal control with distributed model predictive control(DMPC)optimization and introduces a unique distributed attack detector to ensure the reliability of the transmitted information among vehicles.Notably,our strategy uses previously broadcasted information and a specialized convex set,termed the“resilience set”,to identify unreliable data.This approach significantly eases graph robustness prerequisites,requiring only an(F+1)-robust graph,in contrast to the established mean sequence reduced algorithms,which require a minimum(2F+1)-robust graph.Additionally,we introduce a verification algorithm to restore trust in vehicles under minor attacks,further reducing communication network robustness.Our analysis demonstrates the recursive feasibility of the DMPC optimization.Furthermore,the proposed method achieves exceptional control performance by minimizing the discrepancies between the DMPC control inputs and predesigned platoon control inputs,while ensuring constraint compliance and cybersecurity.Simulation results verify the effectiveness of our theoretical findings.

Smart prediction of liquefaction-induced lateral spreading

The prediction of liquefaction-induced lateral spreading/displacement(Dh)is a challenging task for civil/geotechnical engineers.In this study,a new approach is proposed to predict Dh using gene expression programming(GEP).Based on statistical reasoning,individual models were developed for two topographies:free-face and gently sloping ground.Along with a comparison with conventional approaches for predicting the Dh,four additional regression-based soft computing models,i.e.Gaussian process regression(GPR),relevance vector machine(RVM),sequential minimal optimization regression(SMOR),and M5-tree,were developed and compared with the GEP model.The results indicate that the GEP models predict Dh with less bias,as evidenced by the root mean square error(RMSE)and mean absolute error(MAE)for training(i.e.1.092 and 0.815;and 0.643 and 0.526)and for testing(i.e.0.89 and 0.705;and 0.773 and 0.573)in free-face and gently sloping ground topographies,respectively.The overall performance for the free-face topology was ranked as follows:GEP>RVM>M5-tree>GPR>SMOR,with a total score of 40,32,24,15,and 10,respectively.For the gently sloping condition,the performance was ranked as follows:GEP>RVM>GPR>M5-tree>SMOR with a total score of 40,32,21,19,and 8,respectively.Finally,the results of the sensitivity analysis showed that for both free-face and gently sloping ground,the liquefiable layer thickness(T_(15))was the major parameter with percentage deterioration(%D)value of 99.15 and 90.72,respectively.

Label Recovery and Trajectory Designable Network for Transfer Fault Diagnosis of Machines With Incorrect Annotation

The success of deep transfer learning in fault diagnosis is attributed to the collection of high-quality labeled data from the source domain.However,in engineering scenarios,achieving such high-quality label annotation is difficult and expensive.The incorrect label annotation produces two negative effects:1)the complex decision boundary of diagnosis models lowers the generalization performance on the target domain,and2)the distribution of target domain samples becomes misaligned with the false-labeled samples.To overcome these negative effects,this article proposes a solution called the label recovery and trajectory designable network(LRTDN).LRTDN consists of three parts.First,a residual network with dual classifiers is to learn features from cross-domain samples.Second,an annotation check module is constructed to generate a label anomaly indicator that could modify the abnormal labels of false-labeled samples in the source domain.With the training of relabeled samples,the complexity of diagnosis model is reduced via semi-supervised learning.Third,the adaptation trajectories are designed for sample distributions across domains.This ensures that the target domain samples are only adapted with the pure-labeled samples.The LRTDN is verified by two case studies,in which the diagnosis knowledge of bearings is transferred across different working conditions as well as different yet related machines.The results show that LRTDN offers a high diagnosis accuracy even in the presence of incorrect annotation.

IRS Assisted UAV Communications against Proactive Eavesdropping in Mobile Edge Computing Networks

In this paper,we consider mobile edge computing(MEC)networks against proactive eavesdropping.To maximize the transmission rate,IRS assisted UAV communications are applied.We take the joint design of the trajectory of UAV,the transmitting beamforming of users,and the phase shift matrix of IRS.The original problem is strong non-convex and difficult to solve.We first propose two basic modes of the proactive eavesdropper,and obtain the closed-form solution for the boundary conditions of the two modes.Then we transform the original problem into an equivalent one and propose an alternating optimization(AO)based method to obtain a local optimal solution.The convergence of the algorithm is illustrated by numerical results.Further,we propose a zero forcing(ZF)based method as sub-optimal solution,and the simulation section shows that the proposed two schemes could obtain better performance compared with traditional schemes.

Digitalization and intelligentization of manufacturing industry

1 Relying on innovation to realize the historical leapChina has entered a new historical period in her development. In order to achieve scientific development and to accelerate transformation of economic development pattern, the most fundamental issue is to rely on the power of science and technology, and the most crucial element is to improve the capability of independent innovation. The key for promoting China＇s economic and social development is to embark on the innovation-driven track as soon as possible.

Current status and construction scheme of smart geothermal field technology

To address the key problems in the application of intelligent technology in geothermal development,smart application scenarios for geothermal development are constructed.The research status and existing challenges of intelligent technology in each scenario are analyzed,and the construction scheme of smart geothermal field system is proposed.The smart geothermal field is an organic integration of geothermal development engineering and advanced technologies such as the artificial intelligence.At present,the technology of smart geothermal field is still in the exploratory stage.It has been tested for application in scenarios such as intelligent characterization of geothermal reservoirs,dynamic intelligent simulation of geothermal reservoirs,intelligent optimization of development schemes and smart management of geothermal development.However,it still faces many problems,including the high computational cost,difficult real-time response,multiple solutions and strong model dependence,difficult real-time optimization of dynamic multi-constraints,and deep integration of multi-source data.The construction scheme of smart geothermal field system is proposed,which consists of modules including the full database,intelligent characterization,intelligent simulation and intelligent optimization control.The connection between modules is established through the data transmission and the model interaction.In the next stage,it is necessary to focus on the basic theories and key technologies in each module of the smart geothermal field system,to accelerate the lifecycle intelligent transformation of the geothermal development and utilization,and to promote the intelligent,stable,long-term,optimal and safe production of geothermal resources.

Multi-level intelligence empowering lithium-ion batteries

With the significant and widespread application of lithium-ion batteries,there is a growing demand for improved performances of lithium-ion batteries.The intricate degradation throughout the whole lifecycle profoundly impacts the safety,durability,and reliability of lithium-ion batteries.To ensure the long-term,safe,and efficient operation of lithium-ion batteries in various fields,there is a pressing need for enhanced battery intelligence that can withstand extreme events.This work reviews the current status of intelligent battery technology from three perspectives:intelligent response,intelligent sensing,and intelligent management.The intelligent response of battery materials forms the foundation for battery stability,the intelligent sensing of multi-dimensional signals is essential for battery management,and the intelligent management ensures the long-term stable operation of lithium-ion batteries.The critical challenges encountered in the development of intelligent battery technology from each perspective are thoroughly analyzed,and potential solutions are proposed,aiming to facilitate the rapid development of intelligent battery technologies.

Intelligent casting:Empowering the future foundry industry

Emerging technological advances are reshaping the casting sector in latest decades.Casting technology is evolving towards intelligent casting paradigm that involves automation,greenization and intelligentization,which attracts more and more attention from the academic and industry communities.In this paper,the main features of casting technology were briefly summarized and forecasted,and the recent developments of key technologies and the innovative efforts made in promoting intelligent casting process were discussed.Moreover,the technical visions of intelligent casting process were also put forward.The key technologies for intelligent casting process comprise 3D printing technologies,intelligent mold technologies and intelligent process control technologies.In future,the intelligent mold that derived from mold with sensors,control devices and actuators will probably incorporate the Internet of Things,online inspection,embedded simulation,decision-making and control system,and other technologies to form intelligent cyber-physical casting system,which may pave the way to realize intelligent casting.It is promising that the intelligent casting process will eventually achieve the goal of real-time process optimization and full-scale control,with the defects,microstructure,performance,and service life of the fabricated castings can be accurately predicted and tailored.

A real-time intelligent lithology identification method based on a dynamic felling strategy weighted random forest algorithm

Real-time intelligent lithology identification while drilling is vital to realizing downhole closed-loop drilling. The complex and changeable geological environment in the drilling makes lithology identification face many challenges. This paper studies the problems of difficult feature information extraction,low precision of thin-layer identification and limited applicability of the model in intelligent lithologic identification. The author tries to improve the comprehensive performance of the lithology identification model from three aspects: data feature extraction, class balance, and model design. A new real-time intelligent lithology identification model of dynamic felling strategy weighted random forest algorithm(DFW-RF) is proposed. According to the feature selection results, gamma ray and 2 MHz phase resistivity are the logging while drilling(LWD) parameters that significantly influence lithology identification. The comprehensive performance of the DFW-RF lithology identification model has been verified in the application of 3 wells in different areas. By comparing the prediction results of five typical lithology identification algorithms, the DFW-RF model has a higher lithology identification accuracy rate and F1 score. This model improves the identification accuracy of thin-layer lithology and is effective and feasible in different geological environments. The DFW-RF model plays a truly efficient role in the realtime intelligent identification of lithologic information in closed-loop drilling and has greater applicability, which is worthy of being widely used in logging interpretation.

Distributed IRS-Aided DF Relaying Systems:Performance Analysis and Optimization

Intelligent reflecting surface(IRS)is a newly emerged and promising paradigm to substantially improve the performance of wireless communications by constructing favorable communication channels via properly tuning massive reflecting elements.This paper considers a distributed IRS aided decode-and-forward(DF)relaying system over Nakagami-m fading channels.Based on a tight approximation for the distribution of the received signalto-noise ratio(SNR),we first derive exact closed-form expressions of the outage probability,ergodic capacity,and energy efficiency for the considered system.Moreover,we propose the optimal IRS configuration considering the energy efficiency and pilot overhead.Finally,we compare the performance between the distributed IRS-aided DF relaying and multi-IRS-only systems,and verify the analytical results by using monte carlo simulations.

Introduction to the Special Issue on Machine Learning-Guided Intelligent Modeling with Its Industrial Applications

With the advancement of Artificial Intelligence(AI)technology,traditional industrial systems are undergoing an intelligent transformation,bringing together advanced computing,communication and control technologies,Machine Learning(ML)-based intelligentmodelling has become a newparadigm for solving problems in the industrial domain[1–3].With numerous applications and diverse data types in the industrial domain,algorithmic and data-driven ML techniques can intelligently learn potential correlations between complex data and make efficient decisions while reducing human intervention.However,in real-world application scenarios,existing algorithms may have a variety of limitations,such as small data volumes,small detection targets,low efficiency,and algorithmic gaps in specific application domains[4].Therefore,many new algorithms and strategies have been proposed to address the challenges in industrial applications[5–8].

A novel small-animal locomotor activity recording device for biological clock research

The rodent running-wheel recording apparatus is a reliable approach for studying cir-cadian rhythm.This study demonstrated how to construct a simple and intelligent running-wheel recording system.The running wheel was attached to the cage's base,whereas the Hall sensor was attached to the cage's cover.Then,the RJ25 adaptor relayed the running signal to the main control board.Finally,the main control board was connected to the USB port of the computer with the USB connection.Data were collected using the online-accessible,self-created software Magturning.Through Magturning,generated data were saved and exported in real time.Afterward,the device was validated by collecting data on the locomotor activities of mice under dif-ferent light conditions.In conclusion,this new device can record circadian activity of rodents.Our device is appropriate for interdisciplinary investigations related to biological clock research.

Safety control strategy of spinal lamina cutting based on force and cutting depth signals

Laminectomy is one of the most common posterior spinal operations. Since the lamina is adjacent to important tissues such as nerves, once damaged, it can cause serious com-plications and even lead to paralysis. In order to prevent the above injuries and com-plications, ultrasonic bone scalpel and surgical robots have been introduced into spinal laminectomy, and many scholars have studied the recognition method of the bone tissue status. Currently, almost all methods to achieve recognition of bone tissue are based on sensor signals collected by high‐precision sensors installed at the end of surgical robots. However, the previous methods could not accurately identify the state of spinal bone tissue. Innovatively, the identification of bone tissue status was regarded as a time series classification task, and the classification algorithm LSTM‐FCN was used to process fusion signals composed of force and cutting depth signals, thus achieving an accurate classi-fication of the lamina bone tissue status. In addition, it was verified that the accuracy of the proposed method could reach 98.85% in identifying the state of porcine spinal laminectomy. And the maximum penetration distance can be controlled within 0.6 mm, which is safe and can be used in practice.

Improving the Segmentation of Arabic Handwriting Using Ligature Detection Technique

Recognizing handwritten characters remains a critical and formidable challenge within the realm of computervision. Although considerable strides have been made in enhancing English handwritten character recognitionthrough various techniques, deciphering Arabic handwritten characters is particularly intricate. This complexityarises from the diverse array of writing styles among individuals, coupled with the various shapes that a singlecharacter can take when positioned differently within document images, rendering the task more perplexing. Inthis study, a novel segmentation method for Arabic handwritten scripts is suggested. This work aims to locatethe local minima of the vertical and diagonal word image densities to precisely identify the segmentation pointsbetween the cursive letters. The proposed method starts with pre-processing the word image without affectingits main features, then calculates the directions pixel density of the word image by scanning it vertically and fromangles 30° to 90° to count the pixel density fromall directions and address the problem of overlapping letters, whichis a commonly attitude in writing Arabic texts by many people. Local minima and thresholds are also determinedto identify the ideal segmentation area. The proposed technique is tested on samples obtained fromtwo datasets: Aself-curated image dataset and the IFN/ENIT dataset. The results demonstrate that the proposed method achievesa significant improvement in the proportions of cursive segmentation of 92.96% on our dataset, as well as 89.37%on the IFN/ENIT dataset.

Microseismic source location using deep learning:A coal mine case study in China

Microseismic source location is crucial for the early warning of rockburst risks.However,the conventional methods face challenges in terms of the microseismic wave velocity and arrival time accuracy.Intelligent techniques,such as the full convolutional neural network(FCNN),can capture spatial information but struggle with complex microseismic sequence.Combining the FCNN with the long shortterm memory(LSTM)network enables better time-series signal classification by integrating multiscale information and is therefore suitable for waveform location.The LSTM-FCNN model does not require extensive data preprocessing and it simplifies the microseismic source location through feature extraction.In this study,we utilized the LSTM-FCNN as a regression learning model to locate the seismic focus.Initially,the method of short-time-average/long-time-average(STA/LTA)arrival time picking was employed to augment spatiotemporal information.Subsequently,oversampling the on-site data was performed to address the issue of data imbalance,and finally,the performance of LSTM-FCNN was tested.Meanwhile,we compared the LSTM-FCNN model with previous deep-learning models.Our results demonstrated remarkable location capabilities with a mean absolute error(MAE)of only 7.16 m.The model can realize swift training and high accuracy,thereby significantly improving risk warning of rockbursts.

An intelligent control method based on artificial neural network for numerical flight simulation of the basic finner projectile with pitching maneuver

In this paper,an intelligent control method applying on numerical virtual flight is proposed.The proposed algorithm is verified and evaluated by combining with the case of the basic finner projectile model and shows a good application prospect.Firstly,a numerical virtual flight simulation model based on overlapping dynamic mesh technology is constructed.In order to verify the accuracy of the dynamic grid technology and the calculation of unsteady flow,a numerical simulation of the basic finner projectile without control is carried out.The simulation results are in good agreement with the experiment data which shows that the algorithm used in this paper can also be used in the design and evaluation of the intelligent controller in the numerical virtual flight simulation.Secondly,combined with the real-time control requirements of aerodynamic,attitude and displacement parameters of the projectile during the flight process,the numerical simulations of the basic finner projectile’s pitch channel are carried out under the traditional PID(Proportional-Integral-Derivative)control strategy and the intelligent PID control strategy respectively.The intelligent PID controller based on BP(Back Propagation)neural network can realize online learning and self-optimization of control parameters according to the acquired real-time flight parameters.Compared with the traditional PID controller,the concerned control variable overshoot,rise time,transition time and steady state error and other performance indicators have been greatly improved,and the higher the learning efficiency or the inertia coefficient,the faster the system,the larger the overshoot,and the smaller the stability error.The intelligent control method applying on numerical virtual flight is capable of solving the complicated unsteady motion and flow with the intelligent PID control strategy and has a strong promotion to engineering application.

Vision based intelligent traffic light management system using Faster R‐CNN

Transportation systems primarily depend on vehicular flow on roads. Developed coun-tries have shifted towards automated signal control, which manages and updates signal synchronisation automatically. In contrast, traffic in underdeveloped countries is mainly governed by manual traffic light systems. These existing manual systems lead to numerous issues, wasting substantial resources such as time, energy, and fuel, as they cannot make real‐time decisions. In this work, we propose an algorithm to determine traffic signal durations based on real‐time vehicle density, obtained from live closed circuit television camera feeds adjacent to traffic signals. The algorithm automates the traffic light system, making decisions based on vehicle density and employing Faster R‐CNN for vehicle detection. Additionally, we have created a local dataset from live streams of Punjab Safe City cameras in collaboration with the local police authority. The proposed algorithm achieves a class accuracy of 96.6% and a vehicle detection accuracy of 95.7%. Across both day and night modes, our proposed method maintains an average precision, recall, F1 score, and vehicle detection accuracy of 0.94, 0.98, 0.96 and 0.95, respectively. Our proposed work surpasses all evaluation metrics compared to state‐of‐the‐art methodologies.