Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Traffic Control Based on Integrated Kalman Filtering and Adaptive Quantized Q-Learning Framework for Internet of Vehicles

Intelligent traffic control requires accurate estimation of the road states and incorporation of adaptive or dynamically adjusted intelligent algorithms for making the decision.In this article,these issues are handled by proposing a novel framework for traffic control using vehicular communications and Internet of Things data.The framework integrates Kalman filtering and Q-learning.Unlike smoothing Kalman filtering,our data fusion Kalman filter incorporates a process-aware model which makes it superior in terms of the prediction error.Unlike traditional Q-learning,our Q-learning algorithm enables adaptive state quantization by changing the threshold of separating low traffic from high traffic on the road according to the maximum number of vehicles in the junction roads.For evaluation,the model has been simulated on a single intersection consisting of four roads:east,west,north,and south.A comparison of the developed adaptive quantized Q-learning(AQQL)framework with state-of-the-art and greedy approaches shows the superiority of AQQL with an improvement percentage in terms of the released number of vehicles of AQQL is 5%over the greedy approach and 340%over the state-of-the-art approach.Hence,AQQL provides an effective traffic control that can be applied in today’s intelligent traffic system.

Urban Traffic Control Meets Decision Recommendation System:A Survey and Perspective

Urban traffic control is a multifaceted and demanding task that necessitates extensive decision-making to ensure the safety and efficiency of urban transportation systems.Traditional approaches require traffic signal professionals to manually intervene on traffic control devices at the intersection level,utilizing their knowledge and expertise.However,this process is cumbersome,labor-intensive,and cannot be applied on a large network scale.Recent studies have begun to explore the applicability of recommendation system for urban traffic control,which offer increased control efficiency and scalability.Such a decision recommendation system is complex,with various interdependent components,but a systematic literature review has not yet been conducted.In this work,we present an up-to-date survey that elucidates all the detailed components of a recommendation system for urban traffic control,demonstrates the utility and efficacy of such a system in the real world using data and knowledgedriven approaches,and discusses the current challenges and potential future directions of this field.

Deep learning models semi-automatic training system for quality control of transthoracic echocardiography

Objective To explore the value of deep learning(DL)models semi-automatic training system for automatic optimization of clinical image quality control of transthoracic echocardiography(TTE).Methods Totally 1250 TTE videos from 402 patients were retrospectively collected,including 490 apical four chamber(A4C),310 parasternal long axis view of left ventricle(PLAX)and 450 parasternal short axis view of great vessel(PSAX GV).The videos were divided into development set(245 A4C,155 PLAX,225 PSAX GV),semi-automated training set(98 A4C,62 PLAX,90 PSAX GV)and test set(147 A4C,93 PLAX,135 PSAX GV)at the ratio of 5∶2∶3.Based on development set and semi-automatic training set,DL model of quality control was semi-automatically iteratively optimized,and a semi-automatic training system was constructed,then the efficacy of DL models for recognizing TTE views and assessing imaging quality of TTE were verified in test set.Results After optimization,the overall accuracy,precision,recall,and F1 score of DL models for recognizing TTE views in test set improved from 97.33%,97.26%,97.26%and 97.26%to 99.73%,99.65%,99.77%and 99.71%,respectively,while the overall accuracy for assessing A4C,PLAX and PSAX GV TTE as standard views in test set improved from 89.12%,83.87%and 90.37%to 93.20%,90.32%and 93.33%,respectively.Conclusion The developed DL models semi-automatic training system could improve the efficiency of clinical imaging quality control of TTE and increase iteration speed.

An intelligent control method based on artificial neural network for numerical flight simulation of the basic finner projectile with pitching maneuver

In this paper,an intelligent control method applying on numerical virtual flight is proposed.The proposed algorithm is verified and evaluated by combining with the case of the basic finner projectile model and shows a good application prospect.Firstly,a numerical virtual flight simulation model based on overlapping dynamic mesh technology is constructed.In order to verify the accuracy of the dynamic grid technology and the calculation of unsteady flow,a numerical simulation of the basic finner projectile without control is carried out.The simulation results are in good agreement with the experiment data which shows that the algorithm used in this paper can also be used in the design and evaluation of the intelligent controller in the numerical virtual flight simulation.Secondly,combined with the real-time control requirements of aerodynamic,attitude and displacement parameters of the projectile during the flight process,the numerical simulations of the basic finner projectile’s pitch channel are carried out under the traditional PID(Proportional-Integral-Derivative)control strategy and the intelligent PID control strategy respectively.The intelligent PID controller based on BP(Back Propagation)neural network can realize online learning and self-optimization of control parameters according to the acquired real-time flight parameters.Compared with the traditional PID controller,the concerned control variable overshoot,rise time,transition time and steady state error and other performance indicators have been greatly improved,and the higher the learning efficiency or the inertia coefficient,the faster the system,the larger the overshoot,and the smaller the stability error.The intelligent control method applying on numerical virtual flight is capable of solving the complicated unsteady motion and flow with the intelligent PID control strategy and has a strong promotion to engineering application.

Audio-Text Multimodal Speech Recognition via Dual-Tower Architecture for Mandarin Air Traffic Control Communications

In air traffic control communications (ATCC), misunderstandings between pilots and controllers could result in fatal aviation accidents. Fortunately, advanced automatic speech recognition technology has emerged as a promising means of preventing miscommunications and enhancing aviation safety. However, most existing speech recognition methods merely incorporate external language models on the decoder side, leading to insufficient semantic alignment between speech and text modalities during the encoding phase. Furthermore, it is challenging to model acoustic context dependencies over long distances due to the longer speech sequences than text, especially for the extended ATCC data. To address these issues, we propose a speech-text multimodal dual-tower architecture for speech recognition. It employs cross-modal interactions to achieve close semantic alignment during the encoding stage and strengthen its capabilities in modeling auditory long-distance context dependencies. In addition, a two-stage training strategy is elaborately devised to derive semantics-aware acoustic representations effectively. The first stage focuses on pre-training the speech-text multimodal encoding module to enhance inter-modal semantic alignment and aural long-distance context dependencies. The second stage fine-tunes the entire network to bridge the input modality variation gap between the training and inference phases and boost generalization performance. Extensive experiments demonstrate the effectiveness of the proposed speech-text multimodal speech recognition method on the ATCC and AISHELL-1 datasets. It reduces the character error rate to 6.54% and 8.73%, respectively, and exhibits substantial performance gains of 28.76% and 23.82% compared with the best baseline model. The case studies indicate that the obtained semantics-aware acoustic representations aid in accurately recognizing terms with similar pronunciations but distinctive semantics. The research provides a novel modeling paradigm for semantics-aware speech recognition in air traffic control communications, which could contribute to the advancement of intelligent and efficient aviation safety management.

Regional Multi-Agent Cooperative Reinforcement Learning for City-Level Traffic Grid Signal Control

This article studies the effective traffic signal control problem of multiple intersections in a city-level traffic system.A novel regional multi-agent cooperative reinforcement learning algorithm called RegionSTLight is proposed to improve the traffic efficiency.Firstly a regional multi-agent Q-learning framework is proposed,which can equivalently decompose the global Q value of the traffic system into the local values of several regions Based on the framework and the idea of human-machine cooperation,a dynamic zoning method is designed to divide the traffic network into several strong-coupled regions according to realtime traffic flow densities.In order to achieve better cooperation inside each region,a lightweight spatio-temporal fusion feature extraction network is designed.The experiments in synthetic real-world and city-level scenarios show that the proposed RegionS TLight converges more quickly,is more stable,and obtains better asymptotic performance compared to state-of-theart models.

Cluster DetectionMethod of Endogenous Security Abnormal Attack Behavior in Air Traffic Control Network

In order to enhance the accuracy of Air Traffic Control(ATC)cybersecurity attack detection,in this paper,a new clustering detection method is designed for air traffic control network security attacks.The feature set for ATC cybersecurity attacks is constructed by setting the feature states,adding recursive features,and determining the feature criticality.The expected information gain and entropy of the feature data are computed to determine the information gain of the feature data and reduce the interference of similar feature data.An autoencoder is introduced into the AI(artificial intelligence)algorithm to encode and decode the characteristics of ATC network security attack behavior to reduce the dimensionality of the ATC network security attack behavior data.Based on the above processing,an unsupervised learning algorithm for clustering detection of ATC network security attacks is designed.First,determine the distance between the clustering clusters of ATC network security attack behavior characteristics,calculate the clustering threshold,and construct the initial clustering center.Then,the new average value of all feature objects in each cluster is recalculated as the new cluster center.Second,it traverses all objects in a cluster of ATC network security attack behavior feature data.Finally,the cluster detection of ATC network security attack behavior is completed by the computation of objective functions.The experiment took three groups of experimental attack behavior data sets as the test object,and took the detection rate,false detection rate and recall rate as the test indicators,and selected three similar methods for comparative test.The experimental results show that the detection rate of this method is about 98%,the false positive rate is below 1%,and the recall rate is above 97%.Research shows that this method can improve the detection performance of security attacks in air traffic control network.

Intelligent PID Control Method for Quadrotor UAV with Serial Humanoid Intelligence

Quadrotor unmanned aerial vehicles(UAVs)are widely used in inspection,agriculture,express delivery,and other fields owing to their low cost and high flexibility.However,the current UAV control system has shortcomings such as poor control accuracy and weak anti-interference ability to a certain extent.To address the control problem of a four-rotor UAV,we propose a method to enhance the controller’s accuracy by considering underactuated dynamics,nonlinearities,and external disturbances.A mathematical model is constructed based on the flight principles of the quadrotor UAV.We develop a control algorithm that combines humanoid intelligence with a cascade Proportional-Integral-Derivative(PID)approach.This algorithm incorporates the rate of change of the error into the inputs of the cascade PID controller,uses both the error and its rate of change as characteristic variables of the UAV’s control system,and employs a hyperbolic tangent function to improve the outer-loop control.The result is a double closed-loop intelligent PID(DCLIPID)control algorithm.Through MATLAB numerical simulation tests,it is found that the DCLIPID algorithm reduces the rise time by 0.5 s and the number of oscillations by 2 times compared to the string PID algorithm when a unit step signal is used as input.A UAV flight test was designed for comparison with the serial PID algorithm,and it was found that when the UAV planned the trajectory autonomously,the errors in the X-,Y-,and Z-directions were reduced by 0.22,0.21,and 0.31 m,respectively.Under the interference environment of artificial wind about 3.6 m·s^(-1),the UAV hovering error in X-,Y-,and Z-directions are 0.24,0.42,and 0.27 m,respectively.The simulation and experimental results show that the control method of humanoid intelligence and cascade PID can improve the real-time,control accuracy and anti-interference ability of the UAV,and the method has a certain reference value for the research in the field of UAV control.

Intelligent Fractional-Order Controller for SMES Systems in Renewable Energy-Based Microgrid

An autonomous microgrid that runs on renewable energy sources is presented in this article.It has a supercon-ducting magnetic energy storage(SMES)device,wind energy-producing devices,and an energy storage battery.However,because such microgrids are nonlinear and the energy they create varies with time,controlling and managing the energy inside them is a difficult issue.Fractional-order proportional integral(FOPI)controller is recommended for the current research to enhance a standalone microgrid’s energy management and performance.The suggested dedicated control for the SMES comprises two loops:the outer loop,which uses the FOPI to regulate the DC-link voltage,and the inner loop,responsible for regulating the SMES current,is constructed using the intelligent FOPI(iFOPI).The FOPI+iFOPI parameters are best developed using the dandelion optimizer(DO)approach to achieve the optimum performance.The suggested FOPI+iFOPI controller’s performance is contrasted with a conventional PI controller for variations in wind speed and microgrid load.The optimal FOPI+iFOPI controller manages the voltage and frequency of the load.The behavior of the microgrid as a reaction to step changes in load and wind speed was measured using the proposed controller.MATLAB simulations were used to evaluate the recommended system’s performance.The results of the simulations showed that throughout all interruptions,the recommended microgrid provided the load with AC power with a constant amplitude and frequency.In addition,the required load demand was accurately reduced.Furthermore,the microgrid functioned incredibly well despite SMES and varying wind speeds.Results obtained under identical conditions were compared with and without the best FOPI+iFOPI controller.When utilizing the optimal FOPI+iFOPI controller with SMES,it was found that the microgrid performed better than the microgrid without SMES.

Enhancing Patients Outcomes and Infection Control through Smart Indoor Air Quality Monitoring Systems

Air pollution poses a critical threat to public health and environmental sustainability globally, and Nigeria is no exception. Despite significant economic growth and urban development, Nigeria faces substantial air quality challenges, particularly in urban centers. While outdoor air pollution has received considerable attention, the issue of indoor air quality remains underexplored yet equally critical. This study aims to develop a reliable, cost-effective, and user-friendly solution for continuous monitoring and reporting of indoor air quality, accessible from anywhere via a web interface. Addressing the urgent need for effective indoor air quality monitoring in urban hospitals, the research focuses on designing and implementing a smart indoor air quality monitoring system using Arduino technology. Employing an Arduino Uno, ESP8266 Wi-Fi module, and MQ135 gas sensor, the system collects real-time air quality data, transmits it to the ThingSpeak cloud platform, and visualizes it through a user-friendly web interface. This project offers a cost-effective, portable, and reliable solution for monitoring indoor air quality, aiming to mitigate health risks and promote a healthier living environment.

Research on the Optimization Control Method of Inbound Traffic Flow on On-ramp

This study aims to optimize the inbound traffic flow on on-ramps by considering low time costs,good speed stability,and high driving safety for mixed traffic flow.The optimal inlet gap is identified in advance,and trajectory guidance for vehicles entering the gap is determined under safety constraints.Based on the initial state and sequence of vehicles entering the merging area,individual vehicle trajectories are optimized sequentially.An optimization model and method for ramp entry trajectories in mixed traffic flow are developed,incorporating on-ramp vehicle entry sequencing and ordinary vehicle trajectory prediction.Key performance indicators,including driving safety,total travel time,parking wait probability,and trajectory smoothness,are compared and analyzed to evaluate the proposed approach.

A self-learning human-machine cooperative control method based on driver intention recognition

Human-machine cooperative control has become an important area of intelligent driving,where driver intention recognition and dynamic control authority allocation are key factors for improving the performance of cooperative decision-making and control.In this paper,an online learning method is proposed for human-machine cooperative control,which introduces a priority control parameter in the reward function to achieve optimal allocation of control authority under different driver intentions and driving safety conditions.Firstly,a two-layer LSTM-based sequence prediction algorithm is proposed to recognise the driver's lane change(LC)intention for human-machine cooperative steering control.Secondly,an online reinforcement learning method is developed for optimising the steering authority to reduce driver workload and improve driving safety.The driver-in-the-loop simulation results show that our method can accurately predict the driver's LC intention in cooperative driving and effectively compensate for the driver's non-optimal driving actions.The experimental results on a real intelligent vehicle further demonstrate the online optimisation capability of the proposed RL-based control authority allocation algorithm and its effectiveness in improving driving safety.

A Lane Change Model Considering the Stability of Cooperative Adaptive Cruise Control Platoon Fleet

In this article,lane change models for mixed traffic flow under cooperative adaptive cruise control(CACC)platoon formation are established.The analysis begins by examining the impact of lane changes on traffic flow stability.The influences of various factors such as lane change locations,timing,and the current traffic state on stability are discussed.In this analysis,it is assumed that the lane change location and the entry position in the adjacent lane have already been selected,without considering the specific intention behind the lane change.The speeds of the involved vehicles are adjusted based on an existing lane change model,and various conditions are analyzed for traffic flow disturbances,including duration,shock amplitude,and driving delays.Numerical calculations are provided to illustrate these effects.Additionally,traffic flow stability is factored into the lane change decision-making process.By incorporating disturbances to the fleet into the lane change income model,both a lane change intention model and a lane change execution model are constructed.These models are then compared with a model that does not account for stability,leading to the corresponding conclusions.

Design and Application of Intelligent Control System for Molten Iron Transportation Based on 5G Technology

Molten transport is an important link in the iron and steel enterprise production,involves many complex factors,artificial management is difficult.Therefore,puts forward a kind of molten iron transport wisdom control system based on 5G technology,which mainly contains the intelligent identification tracking system,equipment status collection information acquisition system,locomotive vehicle terminal system,etc.Combined with the analysis of the actual application situation,the system could integrate all the processes and elements of molten iron produc-tion and transportation,realize the integration of operation and management,and also promote the improvement of the turnover efficiency of molten iron tank,reduce the demand for personnel,and reduce the labor cost.

城市过饱和区域积分终端滑模控制算法

【目标】现有基于滑模变结构的区域边界控制算法存在明显缺陷,即系统状态需要在时间趋于无穷大时才能收敛到平衡点,仅能保证系统的渐进稳定。这显然无法满足现代城市交通管理对于快速响应和高效调控的需求。为克服该局限,提出一种基于宏观基本图和积分终端滑模的区域边界控制算法,获得较快的收敛速度以及较好的鲁棒性和扰动抑制特性。【方法】首先,将干扰(非门控或区域内部交通量)和宏观基本图建模中的不确定性视为集总干扰,建立城市过饱和区域边界控制系统模型。在此基础上,引入非线性积分终端滑动超平面,设计一种基于积分终端滑模的边界控制算法。【结果】将所提方法应用于5×5均匀棋盘路网中,路网累积空间交通流量有超过1%的提升,部分算例中接近5%;平均速度有超过5%的提升,部分算例中超过8%。所提出算法具有较快的收敛速度和较好的扰动抑制能力,能显著提升路网累计吞吐量和路网中车辆平均速度,缓解过饱和区域交通拥堵。【结论】相较于既有滑模变结构边界控制算法,所提出算法采用非线性积分终端滑动超平面,能保证受控路网状态在预设有限时间内收敛到期望状态;相较于既有比例积分控制策略,能更好处理受控路网中非常值干扰交通需求和宏观基本图建模中存在的低离散度。

高精度地图背景下智能网联交通工程课程改革

随着高精度地图和智能网联技术的快速发展,“互联网+”、人工智能、大数据和自动驾驶等新技术不断取得突破,交通运输行业正经历着一场革命性的变革。在高精度地图背景下,将智能网联和专业课程融合,探索面向智能网联交通系统的交通工程专业课程教材改革。为让学生能够更好地了解新一代交通运输系统和进化的规律,针对当前教学模式存在的缺陷,可以从增强课程内容与社会发展的联系、引入与网联交通相关的科技论文及重构课程体系结构等多个方面,对智能网联交通工程课程进行改革和研究,为高精度地图下的课程改革研究提供新思路和新方向。

农用车辆导航系统语音控制功能的实现——基于汉语言声学特征

随着先进技术的应用,农用车辆在农业生产中发挥着重要作用。为此,将汉语言声学特征的语音控制应用在农用车辆导航系统中,通过集成讯飞语音云实现安静和嘈杂两种环境下对导航系统的实时控制。实验结果表明:语音识别正确率很高,平均达到了93.17%,具有较高的实用价值和推广价值。

NSGA-Ⅱ based traffic signal control optimization algorithm for over-saturated intersection group

In order to improve the efficiency of traffic signal control for an over-saturated intersection group, a nondominated sorting genetic algorithm Ⅱ（NSGA-Ⅱ） based traffic signal control optimization algorithm is proposed. The throughput maximum and average queue ratio minimum for the critical route of the intersection group are selected as the optimization objectives of the traffic signal control for the over-saturated condition. The consequences of the efficiency between traffic signal timing plans generated by the proposed algorithm and a commonly utilized signal timing optimization software Synchro are compared in a VISSIM signal control application programming interfaces （SCAPI） simulation environment by using real filed observed traffic data. The simulation results indicate that the signal timing plan generated by the proposed algorithm is more efficient in managing oversaturated flows at intersection groups, and, thus, it has the capability of optimizing signal timing under the over-saturated conditions.

质量快检特性数据采集与分析实验项目设计

该文设计了一个面向质量管理工程专业学生的综合快检实验。实验以便携式光谱快速采集样品质量特性数据,运用质量统计工具和智能学习算法实现数据快速分析和规律揭示。通过拉曼光谱仪照射乳品获得样品的散射特征信号,每个实验样品数据采集仅需60s。选取特征峰值,结合质量统计控制分析工具绘制样品质量控制图,可描述样品质量波动情况。运用极限学习机算法结合数据预处理方法,优化实现样品品牌快速判别,识别率可达97.3%,算法运行时间仅需1s。