Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

An adaptive physics-informed deep learning method for pore pressure prediction using seismic data

Accurate prediction of formation pore pressure is essential to predict fluid flow and manage hydrocarbon production in petroleum engineering.Recent deep learning technique has been receiving more interest due to the great potential to deal with pore pressure prediction.However,most of the traditional deep learning models are less efficient to address generalization problems.To fill this technical gap,in this work,we developed a new adaptive physics-informed deep learning model with high generalization capability to predict pore pressure values directly from seismic data.Specifically,the new model,named CGP-NN,consists of a novel parametric features extraction approach(1DCPP),a stacked multilayer gated recurrent model(multilayer GRU),and an adaptive physics-informed loss function.Through machine training,the developed model can automatically select the optimal physical model to constrain the results for each pore pressure prediction.The CGP-NN model has the best generalization when the physicsrelated metricλ=0.5.A hybrid approach combining Eaton and Bowers methods is also proposed to build machine-learnable labels for solving the problem of few labels.To validate the developed model and methodology,a case study on a complex reservoir in Tarim Basin was further performed to demonstrate the high accuracy on the pore pressure prediction of new wells along with the strong generalization ability.The adaptive physics-informed deep learning approach presented here has potential application in the prediction of pore pressures coupled with multiple genesis mechanisms using seismic data.

Underwater four-quadrant dual-beam circumferential scanning laser fuze using nonlinear adaptive backscatter filter based on pauseable SAF-LMS algorithm

The phenomenon of a target echo peak overlapping with the backscattered echo peak significantly undermines the detection range and precision of underwater laser fuzes.To overcome this issue,we propose a four-quadrant dual-beam circumferential scanning laser fuze to distinguish various interference signals and provide more real-time data for the backscatter filtering algorithm.This enhances the algorithm loading capability of the fuze.In order to address the problem of insufficient filtering capacity in existing linear backscatter filtering algorithms,we develop a nonlinear backscattering adaptive filter based on the spline adaptive filter least mean square(SAF-LMS)algorithm.We also designed an algorithm pause module to retain the original trend of the target echo peak,improving the time discrimination accuracy and anti-interference capability of the fuze.Finally,experiments are conducted with varying signal-to-noise ratios of the original underwater target echo signals.The experimental results show that the average signal-to-noise ratio before and after filtering can be improved by more than31 d B,with an increase of up to 76%in extreme detection distance.

Adaptive Trajectory Tracking Control for Nonholonomic Wheeled Mobile Robots:A Barrier Function Sliding Mode Approach

The trajectory tracking control performance of nonholonomic wheeled mobile robots(NWMRs)is subject to nonholonomic constraints,system uncertainties,and external disturbances.This paper proposes a barrier function-based adaptive sliding mode control(BFASMC)method to provide high-precision,fast-response performance and robustness for NWMRs.Compared with the conventional adaptive sliding mode control,the proposed control strategy can guarantee that the sliding mode variables converge to a predefined neighborhood of origin with a predefined reaching time independent of the prior knowledge of the uncertainties and disturbances bounds.Another advantage of the proposed algorithm is that the control gains can be adaptively adjusted to follow the disturbances amplitudes thanks to the barrier function.The benefit is that the overestimation of control gain can be eliminated,resulting in chattering reduction.Moreover,a modified barrier function-like control gain is employed to prevent the input saturation problem due to the physical limit of the actuator.The stability analysis and comparative experiments demonstrate that the proposed BFASMC can ensure the prespecified convergence performance of the NWMR system output variables and strong robustness against uncertainties/disturbances.

Adaptive Optimal Discrete-Time Output-Feedback Using an Internal Model Principle and Adaptive Dynamic Programming

In order to address the output feedback issue for linear discrete-time systems, this work suggests a brand-new adaptive dynamic programming(ADP) technique based on the internal model principle(IMP). The proposed method, termed as IMP-ADP, does not require complete state feedback-merely the measurement of input and output data. More specifically, based on the IMP, the output control problem can first be converted into a stabilization problem. We then design an observer to reproduce the full state of the system by measuring the inputs and outputs. Moreover, this technique includes both a policy iteration algorithm and a value iteration algorithm to determine the optimal feedback gain without using a dynamic system model. It is important that with this concept one does not need to solve the regulator equation. Finally, this control method was tested on an inverter system of grid-connected LCLs to demonstrate that the proposed method provides the desired performance in terms of both tracking and disturbance rejection.

Automatic modulation recognition of radio fuzes using a DR2D-based adaptive denoising method and textural feature extraction

The identification of intercepted radio fuze modulation types is a prerequisite for decision-making in interference systems.However,the electromagnetic environment of modern battlefields is complex,and the signal-to-noise ratio(SNR)of such environments is usually low,which makes it difficult to implement accurate recognition of radio fuzes.To solve the above problem,a radio fuze automatic modulation recognition(AMR)method for low-SNR environments is proposed.First,an adaptive denoising algorithm based on data rearrangement and the two-dimensional(2D)fast Fourier transform(FFT)(DR2D)is used to reduce the noise of the intercepted radio fuze intermediate frequency(IF)signal.Then,the textural features of the denoised IF signal rearranged data matrix are extracted from the statistical indicator vectors of gray-level cooccurrence matrices(GLCMs),and support vector machines(SVMs)are used for classification.The DR2D-based adaptive denoising algorithm achieves an average correlation coefficient of more than 0.76 for ten fuze types under SNRs of-10 d B and above,which is higher than that of other typical algorithms.The trained SVM classification model achieves an average recognition accuracy of more than 96%on seven modulation types and recognition accuracies of more than 94%on each modulation type under SNRs of-12 d B and above,which represents a good AMR performance of radio fuzes under low SNRs.

An Adaptive Program Recommendation System for Multi-User Sharing Environment

More and more accounts or devices are shared by multiple users in video applications,which makes it difficult to provide recommendation service.Existing recommendation schemes overlook multiuser sharing scenarios,and they cannot make effective use of the mixed information generated by multi-user when exploring users’potential interests.To solve these problems,this paper proposes an adaptive program recommendation system for multi-user sharing environment.Specifically,we first design an offline periodic identification module by building multi-user features and periodically predicting target user in future sessions,which can separate the profile of target user from mixed log records.Subsequently,an online recommendation module with adaptive timevarying exploration strategy is constructed by jointly using personal information and multi-user social information provided by identification module.On one hand,to learn the dynamic changes in user-interest,a time-varying linear upper confidence bound(LinUCB)based on personal information is designed.On the other hand,to reduce the risk of exploration,a timeinvariant LinUCB based on separated multi-user social information from one account/device is proposed to compute the quality scores of programs for each user,which is integrated into the time-varying LinUCB by cross-weighting strategy.Finally,experimental results validate the efficiency of the proposed scheme.

Ground-layer Adaptive Optics for the 2.5 m Wide-field and High-resolution Solar Telescope

The 2.5 m wide-field and high-resolution solar telescope(WeHoST)is currently under developing for solar observations.WeHoST aims to achieve high-resolution observations over a super-wide field of view(FOV)of5′×5′,and a desired resolution of 0.3″.To meet the scientific requirements of WeHoST,the ground-layer adaptive optics(GLAO)with a specially designed wave front sensing system is as the primary consideration.We introduce the GLAO configuration,particularly the wave front sensing scheme.Utilizing analytic method,we simulate the performance of both classical AO and GLAO systems,optimize the wave front sensing system,and evaluate GLAO performance in terms of PSF uniformity and correction improvement across whole FOV.The results indicate that,the classical AO will achieve diffraction-limited resolution;the suggested GLAO configuration will uniformly improve the seeing across the full 5′×5′FOV,reducing the FWHM across the axis FOV to less than0.3″(λ≥705 nm,r0≥11 cm),which is more than two times improvement.The specially designed wave front sensor schedule offers new potential for WeHoST’s GLAO,particularly the multi-FOV GLAO and the flexibility to select the detected area.These capabilities will significantly enhance the scientific output of the telescope.

Tomato detection method using domain adaptive learning for dense planting environments

This study aimed to address the challenge of accurately and reliably detecting tomatoes in dense planting environments,a critical prerequisite for the automation implementation of robotic harvesting.However,the heavy reliance on extensive manually annotated datasets for training deep learning models still poses significant limitations to their application in real-world agricultural production environments.To overcome these limitations,we employed domain adaptive learning approach combined with the YOLOv5 model to develop a novel tomato detection model called as TDA-YOLO(tomato detection domain adaptation).We designated the normal illumination scenes in dense planting environments as the source domain and utilized various other illumination scenes as the target domain.To construct bridge mechanism between source and target domains,neural preset for color style transfer is introduced to generate a pseudo-dataset,which served to deal with domain discrepancy.Furthermore,this study combines the semi-supervised learning method to enable the model to extract domain-invariant features more fully,and uses knowledge distillation to improve the model's ability to adapt to the target domain.Additionally,for purpose of promoting inference speed and low computational demand,the lightweight FasterNet network was integrated into the YOLOv5's C3 module,creating a modified C3_Faster module.The experimental results demonstrated that the proposed TDA-YOLO model significantly outperformed original YOLOv5s model,achieving a mAP(mean average precision)of 96.80%for tomato detection across diverse scenarios in dense planting environments,increasing by 7.19 percentage points;Compared with the latest YOLOv8 and YOLOv9,it is also 2.17 and 1.19 percentage points higher,respectively.The model's average detection time per image was an impressive 15 milliseconds,with a FLOPs(floating point operations per second)count of 13.8 G.After acceleration processing,the detection accuracy of the TDA-YOLO model on the Jetson Xavier NX development board is 90.95%,the mAP value is 91.35%,and the detection time of each image is 21 ms,which can still meet the requirements of real-time detection of tomatoes in dense planting environment.The experimental results show that the proposed TDA-YOLO model can accurately and quickly detect tomatoes in dense planting environment,and at the same time avoid the use of a large number of annotated data,which provides technical support for the development of automatic harvesting systems for tomatoes and other fruits.

Acetic acid-and furfural-based adaptive evolution of Saccharomyces cerevisiae strains for improving stress tolerance and lignocellulosic ethanol production

Acetic acid and furfural are known as prevalent inhibitors deriving from pretreatment during lignocellulosic ethanol production.They negatively impact cell growth,glucose uptake and ethanol biosynthesis of Saccharomyces cerevisiae strains.Development of industrial S.cerevisiae strains with high tolerance towards these inhibitors is thus critical for efficient lignocellulosic ethanol production.In this study,the acetic acid or furfural tolerance of different S.cerevisiae strains could be significantly enhanced after adaptive evolution via serial cultivation for 40 generations under stress conditions.The acetic acid-based adaptive strain SPSC01-TA9 produced 30.5 g·L^(-1)ethanol with a yield of 0.46 g·g^(-1)in the presence of 9 g·L^(-1)acetic acid,while the acetic acid/furfural-based adaptive strain SPSC01-TAF94 produced more ethanol of 36.2 g·L^(-1)with increased yield up to 0.49 g·g^(-1)in the presence of both 9 g·L^(-1)acetic acid and 4 g·L^(-1)furfural.Significant improvements were also observed during non-detoxified corn stover hydrolysate culture by SPSC01-TAF94,which achieved ethanol production and yield of 29.1 g·L^(-1)and 0.49 g·g^(-1),respectively,the growth and fermentation efficiency of acetic acid/furfural-based adaptive strain in hydrolysate was 95%higher than those of wildtype strains,indicating the acetic acid-and furfural-based adaptive evolution strategy could be an effective approach for improving lignocellulosic ethanol production.The adapted strains developed in this study with enhanced tolerance against acetic acid and furfural could be potentially contribute to economically feasible and sustainable lignocellulosic biorefinery.

Nonlinear robust adaptive control for bidirectional stabilization system of all-electric tank with unknown actuator backlash compensation and disturbance estimation

Since backlash nonlinearity is inevitably existing in actuators for bidirectional stabilization system of allelectric tank,it behaves more drastically in high maneuvering environments.In this work,the accurate tracking control for bidirectional stabilization system of moving all-electric tank with actuator backlash and unmodeled disturbance is solved.By utilizing the smooth adaptive backlash inverse model,a nonlinear robust adaptive feedback control scheme is presented.The unknown parameters and unmodelled disturbance are addressed separately through the derived parametric adaptive function and the continuous nonlinear robust term.Because the unknown backlash parameters are updated via adaptive function and the backlash effect can be suppressed successfully by inverse operation,which ensures the system stability.Meanwhile,the system disturbance in the high maneuverable environment can be estimated with the constructed adaptive law online improving the engineering practicality.Finally,Lyapunov-based analysis proves that the developed controller can ensure the tracking error asymptotically converges to zero even with unmodeled disturbance and unknown actuator backlash.Contrast co-simulations and experiments illustrate the advantages of the proposed approach.

Comparison of Adaptive Simulation Observation Experiments of the Heavy Rainfall in South China and Sichuan Basin

This study examines the effectiveness of adaptive observation experiments using the ensemble transformation sensitivity(ETS) method to improve precipitation forecasts during heavy rainfall events in South China and the Sichuan Basin. High-resolution numerical models are employed to simulate adaptive observations. By identifying the sensitive areas of key weather system positions 42 hours before heavy rainfall events, the adaptive observations improve the prediction of jet streams, strong winds, and shear lines, which are essential for accurate heavy rainfall forecasting. This improvement is reflected in both the precipitation structure and location accuracy within the verification region. In South China, targeted observations enhance rainfall predictions by improving water vapor transport. In the Sichuan Basin, adaptive observations refine water vapor transport and adjust vortex dynamics. This research highlights the importance of accurately predicting shear lines and jet streams for forecasting heavy rainfall in these areas. Overall, this study found that adaptive observation enhances the precipitation forecast skills of the structure and location for heavy rainfall in South China and the Sichuan Basin, emphasizing their potential utility in operational numerical weather prediction.

A Fractional-Order Ultra-Local Model-Based Adaptive Neural Network Sliding Mode Control of n-DOF Upper-Limb Exoskeleton With Input Deadzone

This paper proposes an adaptive neural network sliding mode control based on fractional-order ultra-local model for n-DOF upper-limb exoskeleton in presence of uncertainties,external disturbances and input deadzone.Considering the model complexity and input deadzone,a fractional-order ultra-local model is proposed to formulate the original dynamic system for simple controller design.Firstly,the control gain of ultra-local model is considered as a constant.The fractional-order sliding mode technique is designed to stabilize the closed-loop system,while fractional-order time-delay estimation is combined with neural network to estimate the lumped disturbance.Correspondingly,a fractional-order ultra-local model-based neural network sliding mode controller(FO-NNSMC) is proposed.Secondly,to avoid disadvantageous effect of improper gain selection on the control performance,the control gain of ultra-local model is considered as an unknown parameter.Then,the Nussbaum technique is introduced into the FO-NNSMC to deal with the stability problem with unknown gain.Correspondingly,a fractional-order ultra-local model-based adaptive neural network sliding mode controller(FO-ANNSMC) is proposed.Moreover,the stability analysis of the closed-loop system with the proposed method is presented by using the Lyapunov theory.Finally,with the co-simulations on virtual prototype of 7-DOF iReHave upper-limb exoskeleton and experiments on 2-DOF upper-limb exoskeleton,the obtained compared results illustrate the effectiveness and superiority of the proposed method.

Charge adaptive phytochemical-based nanoparticles for eradication of methicillin-resistant staphylococcus aureus biofilms

The intrinsic resistance of MRSA coupled with biofilm antibiotic tolerance challenges the antibiotic treatment of MRSA biofilm infections.Phytochemical-based nanoplatform is a promising emerging approach for treatment of biofilm infection.However,their therapeutic efficacy was restricted by the low drug loading capacity and lack of selectivity.Herein,we constructed a surface charge adaptive phytochemical-based nanoparticle with high isoliquiritigenin(ISL)loading content for effective treatment of MRSA biofilm.A dimeric ISL prodrug(ISL-G2)bearing a lipase responsive ester bond was synthesized,and then encapsulated into the amphiphilic quaternized oligochitosan.The obtained ISL-G2loaded NPs possessed positively charged surface,which allowed cis-aconityl-D-tyrosine(CA-Tyr)binding via electrostatic interaction to obtain ISL-G2@TMDCOS-Tyr NPs.The NPs maintained their negatively charged surface,thus prolonging the blood circulation time.In response to low pH in the biofilms,the fast removal of CA-Tyr led to a shift in their surface charge from negative to positive,which enhanced the accumulation and penetration of NPs in the biofilms.Sequentially,the pH-triggered release of D-tyrosine dispersed the biofilm and lipase-triggered released of ISL effectively kill biofilm MRSA.An in vivo study was performed on a MRSA biofilm infected wound model.This phytochemical-based system led to~2log CFU(>99%)reduction of biofilm MRSA as compared to untreated wound(P<0.001)with negligible biotoxicity in mice.This phytochemical dimer nanoplatform shows great potential for long-term treatment of resistant bacterial infections.

Quaternion-Based Adaptive Trajectory Tracking Control of a Rotor-Missile with Unknown Parameters Identification

This paper investigates the adaptive trajectory tracking control problem and the unknown parameter identification problem of a class of rotor-missiles with parametric system uncertainties.First,considering the uncertainty of structural and aerodynamic parameters,the six-degree-of-freedom(6Do F) nonlinear equations describing the position and attitude dynamics of the rotor-missile are established,respectively,in the inertial and body-fixed reference frames.Next,a hierarchical adaptive trajectory tracking controller that can guarantee closed-loop stability is proposed according to the cascade characteristics of the 6Do F dynamics.Then,a memory-augmented update rule of unknown parameters is proposed by integrating all historical data of the regression matrix.As long as the finitely excited condition is satisfied,the precise identification of unknown parameters can be achieved.Finally,the validity of the proposed trajectory tracking controller and the parameter identification method is proved through Lyapunov stability theory and numerical simulations.

Fast compressed sensing spectral measurement with adaptive gradient multiscale resolution

We propose a fast,adaptive multiscale resolution spectral measurement method based on compressed sensing.The method can apply variable measurement resolution over the entire spectral range to reduce the measurement time by over 75%compared to a global high-resolution measurement.Mimicking the characteristics of the human retina system,the resolution distribution follows the principle of gradually decreasing.The system allows the spectral peaks of interest to be captured dynamically or to be specified a priori by a user.The system was tested by measuring single and dual spectral peaks,and the results of spectral peaks are consistent with those of global high-resolution measurements.

Adaptive Robust Servo Control for Vertical Electric Stabilization System of Tank and Experimental Validation

A tracking stability control problem for the vertical electric stabilization system of moving tank based on adaptive robust servo control is addressed.This paper mainly focuses on two types of possibly fast timevarying but bounded uncertainty within the vertical electric stabilization system:model parameter uncertainty and uncertain nonlinearity.First,the vertical electric stabilization system is constructed as an uncertain nonlinear dynamic system that can reflect the practical mechanics transfer process of the system.Second,the dynamical equation in the form of state space is established by designing the angular tracking error.Third,the comprehensive parameter of system uncertainty is designed to estimate the most conservative effects of uncertainty.Finally,an adaptive robust servo control which can effectively handle the combined effects of complex nonlinearity and uncertainty is proposed.The feasibility of the proposed control strategy under the practical physical condition is validated through the tests on the experimental platform.This paper pioneers the introduction of the internal nonlinearity and uncertainty of the vertical electric stabilization system into the settlement of the tracking stability control problem,and validates the advanced servo control strategy through experiment for the first time.

The prediction of projectile-target intersection for moving tank based on adaptive robust constraint-following control and interval uncertainty analysis

To improve the hit probability of tank at high speed,a prediction method of projectile-target intersection based on adaptive robust constraint-following control and interval uncertainty analysis is proposed.The method proposed provides a novel way to predict the impact point of projectile for moving tank.First,bidirectional stability constraints and stability constraint-following error are constructed using the Udwadia-Kalaba theory,and an adaptive robust constraint-following controller is designed considering uncertainties.Second,the exterior ballistic ordinary differential equation with uncertainties is integrated into the controller,and the pointing control of stability system is extended to the impact-point control of projectile.Third,based on the interval uncertainty analysis method combining Chebyshev polynomial expansion and affine arithmetic,a prediction method of projectile-target intersection is proposed.Finally,the co-simulation experiment is performed by establishing the multi-body system dynamic model of tank and mathematical model of control system.The results demonstrate that the prediction method of projectile-target intersection based on uncertainty analysis can effectively decrease the uncertainties of system,improve the prediction accuracy,and increase the hit probability.The adaptive robust constraint-following control can effectively restrain the uncertainties caused by road excitation and model error.

An adaptive finite-difference method for seismic traveltime modeling based on 3D eikonal equation

3D eikonal equation is a partial differential equation for the calculation of first-arrival traveltimes and has been widely applied in many scopes such as ray tracing,source localization,reflection migration,seismic monitoring and tomographic imaging.In recent years,many advanced methods have been developed to solve the 3D eikonal equation in heterogeneous media.However,there are still challenges for the stable and accurate calculation of first-arrival traveltimes in 3D strongly inhomogeneous media.In this paper,we propose an adaptive finite-difference(AFD)method to numerically solve the 3D eikonal equation.The novel method makes full use of the advantages of different local operators characterizing different seismic wave types to calculate factors and traveltimes,and then the most accurate factor and traveltime are adaptively selected for the convergent updating based on the Fermat principle.Combined with global fast sweeping describing seismic waves propagating along eight directions in 3D media,our novel method can achieve the robust calculation of first-arrival traveltimes with high precision at grid points either near source point or far away from source point even in a velocity model with large and sharp contrasts.Several numerical examples show the good performance of the AFD method,which will be beneficial to many scientific applications.

Imaging of human parafoveal area with large field of view in adaptive optics line scanning ophthalmoscope

The parafoveal area,with its high concentration of photoreceptors andfine retinal capillaries,is crucial for central vision and often exhibits early signs of pathological changes.The current adaptive optics scanning laser ophthalmoscope(AOSLO)provides an excellent tool to acquire accurate and detailed information about the parafoveal area with cellular resolution.However,limited by the scanning speed of two-dimensional scanning,thefield of view(FOV)in the AOSLO system was usually less than or equal to 2,and the stitching for the parafoveal area required dozens of images,which was time-consuming and laborious.Unfortunately,almost half of patients are unable to obtain stitched images because of their poorfixation.To solve this problem,we integrate AO technology with the line-scan imaging method to build an adaptive optics line scanning ophthalmoscope(AOLSO)system with a larger FOV.In the AOLSO,afocal spherical mirrors in pairs are nonplanar arranged and the distance and angle between optical elements are optimized to minimize the aberrations,two cylinder lenses are orthogonally placed before the imaging sensor to stretch the point spread function(PSF)for sufficiently digitizing light energy.Captured human retinal images show the whole parafoveal area with 55FOV,60 Hz frame rate and cellular resolutions.Take advantage of the 5FOV of the AOLSO,only 9 frames of the retina are captured with several minutes to stitch a montage image with an FOV of 99,in which photoreceptor counting is performed within approximately 5eccentricity.The AOLSO system not only provides cellular resolution but also has the capability to capture the parafoveal region in a single frame,which offers great potential for noninvasive studying of the parafoveal area.