Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

A Study of Multimodal Intelligent Adaptive Learning System and Its Pattern of Promoting Learners’Online Learning Engagement

As the field of artificial intelligence continues to evolve,so too does the application of multimodal learning analysis and intelligent adaptive learning systems.This trend has the potential to promote the equalization of educational resources,the intellectualization of educational methods,and the modernization of educational reform,among other benefits.This study proposes a construction framework for an intelligent adaptive learning system that is supported by multimodal data.It provides a detailed explanation of the system’s working principles and patterns,which aim to enhance learners’online engagement in behavior,emotion,and cognition.The study seeks to address the issue of intelligent adaptive learning systems diagnosing learners’learning behavior based solely on learning achievement,to improve learners’online engagement,enable them to master more required knowledge,and ultimately achieve better learning outcomes.

An Intelligent Neural Networks System for Adaptive Learning and Prediction of a Bioreactor Benchmark Process

The adaptive learning and prediction of a highly nonlinear and time-varying bioreactor benchmark process is studied using Neur-On-Line, a graphical tool kit for developing and deploying neural networks in the G2 real time intelligent environment,and a new modified Broyden, Fletcher, Goldfarb, and Shanno （BFGS） quasi-Newton algorithm. The modified BFGS algorithm for the adaptive learning of back propagation （BP） neural networks is developed and embedded into NeurOn-Line by introducing a new search method of learning rate to the full memory BFGS algorithm. Simulation results show that the adaptive learning and prediction neural network system can quicklv track the time-varving and nonlinear behavior of the bioreactor.

XA-GANomaly: An Explainable Adaptive Semi-Supervised Learning Method for Intrusion Detection Using GANomaly

Intrusion detection involves identifying unauthorized network activity and recognizing whether the data constitute an abnormal network transmission.Recent research has focused on using semi-supervised learning mechanisms to identify abnormal network traffic to deal with labeled and unlabeled data in the industry.However,real-time training and classifying network traffic pose challenges,as they can lead to the degradation of the overall dataset and difficulties preventing attacks.Additionally,existing semi-supervised learning research might need to analyze the experimental results comprehensively.This paper proposes XA-GANomaly,a novel technique for explainable adaptive semi-supervised learning using GANomaly,an image anomalous detection model that dynamically trains small subsets to these issues.First,this research introduces a deep neural network(DNN)-based GANomaly for semi-supervised learning.Second,this paper presents the proposed adaptive algorithm for the DNN-based GANomaly,which is validated with four subsets of the adaptive dataset.Finally,this study demonstrates a monitoring system that incorporates three explainable techniques—Shapley additive explanations,reconstruction error visualization,and t-distributed stochastic neighbor embedding—to respond effectively to attacks on traffic data at each feature engineering stage,semi-supervised learning,and adaptive learning.Compared to other single-class classification techniques,the proposed DNN-based GANomaly achieves higher scores for Network Security Laboratory-Knowledge Discovery in Databases and UNSW-NB15 datasets at 13%and 8%of F1 scores and 4.17%and 11.51%for accuracy,respectively.Furthermore,experiments of the proposed adaptive learning reveal mostly improved results over the initial values.An analysis and monitoring system based on the combination of the three explainable methodologies is also described.Thus,the proposed method has the potential advantages to be applied in practical industry,and future research will explore handling unbalanced real-time datasets in various scenarios.

Decision-Making Models Based on Meta-Reinforcement Learning for Intelligent Vehicles at Urban Intersections

Behavioral decision-making at urban intersections is one of the primary difficulties currently impeding the development of intelligent vehicle technology.The problem is that existing decision-making algorithms cannot effectively deal with complex random scenarios at urban intersections.To deal with this,a deep deterministic policy gradient(DDPG)decision-making algorithm(T-DDPG)based on a time-series Markov decision process(T-MDP)was developed,where the state was extended to collect observations from several consecutive frames.Experiments found that T-DDPG performed better in terms of convergence and generalizability in complex intersection scenarios than a traditional DDPG algorithm.Furthermore,model-agnostic meta-learning(MAML)was incorporated into the T-DDPG algorithm to improve the training method,leading to a decision algorithm(T-MAML-DDPG)based on a secondary gradient.Simulation experiments of intersection scenarios were carried out on the Gym-Carla platform to verify and compare the decision models.The results showed that T-MAML-DDPG was able to easily deal with the random states of complex intersection scenarios,which could improve traffic safety and efficiency.The above decision-making models based on meta-reinforcement learning are significant for enhancing the decision-making ability of intelligent vehicles at urban intersections.

Intelligent decision-making method of TBM operating parameters based on multiple constraints and objective optimization

The decision-making method of tunnel boring machine(TBM)operating parameters has a significant guiding significance for TBM safe and efficient construction,and it has been one of the TBM tunneling research hotspots.For this purpose,this paper introduces an intelligent decision-making method of TBM operating parameters based on multiple constraints and objective optimization.First,linear cutting tests and numerical simulations are used to investigate the physical rules between different cutting parameters(penetration,cutter spacing,etc.)and rock compressive strength.Second,a dual-driven mapping of rock parameters and TBM operating parameters based on data mining and physical rules of rock breaking is established with high accuracy by combining rock-breaking rules and deep neural networks(DNNs).The decision-making method is established by dual-driven mapping,using the effective rock-breaking capacity and the rated value of mechanical parameters as constraints and the total excavation cost as the optimization objective.The best operational parameters can be obtained by searching for the revolutions per minute and penetration that correspond to the extremum of the constrained objective function.The practicability and effectiveness of the developed decision-making model is verified in the SecondWater Source Channel of Hangzhou,China,resulting in the average penetration rate increasing by 11.3%and the total cost decreasing by 10%.

MADDPG-D2: An Intelligent Dynamic Task Allocation Algorithm Based on Multi-Agent Architecture Driven by Prior Knowledge

Aiming at the problems of low solution accuracy and high decision pressure when facing large-scale dynamic task allocation(DTA)and high-dimensional decision space with single agent,this paper combines the deep reinforce-ment learning(DRL)theory and an improved Multi-Agent Deep Deterministic Policy Gradient(MADDPG-D2)algorithm with a dual experience replay pool and a dual noise based on multi-agent architecture is proposed to improve the efficiency of DTA.The algorithm is based on the traditional Multi-Agent Deep Deterministic Policy Gradient(MADDPG)algorithm,and considers the introduction of a double noise mechanism to increase the action exploration space in the early stage of the algorithm,and the introduction of a double experience pool to improve the data utilization rate;at the same time,in order to accelerate the training speed and efficiency of the agents,and to solve the cold-start problem of the training,the a priori knowledge technology is applied to the training of the algorithm.Finally,the MADDPG-D2 algorithm is compared and analyzed based on the digital battlefield of ground and air confrontation.The experimental results show that the agents trained by the MADDPG-D2 algorithm have higher win rates and average rewards,can utilize the resources more reasonably,and better solve the problem of the traditional single agent algorithms facing the difficulty of solving the problem in the high-dimensional decision space.The MADDPG-D2 algorithm based on multi-agent architecture proposed in this paper has certain superiority and rationality in DTA.

Explainable, Domain-Adaptive, and Federated Artificial Intelligence in Medicine

Artificial intelligence(AI)continues to transform data analysis in many domains.Progress in each domain is driven by a growing body of annotated data,increased computational resources,and technological innovations.In medicine,the sensitivity of the data,the complexity of the tasks,the potentially high stakes,and a requirement of accountability give rise to a particular set of challenges.In this review,we focus on three key methodological approaches that address some of the particular challenges in AI-driven medical decision making.1)Explainable AI aims to produce a human-interpretable justification for each output.Such models increase confidence if the results appear plausible and match the clinicians expectations.However,the absence of a plausible explanation does not imply an inaccurate model.Especially in highly non-linear,complex models that are tuned to maximize accuracy,such interpretable representations only reflect a small portion of the justification.2)Domain adaptation and transfer learning enable AI models to be trained and applied across multiple domains.For example,a classification task based on images acquired on different acquisition hardware.3)Federated learning enables learning large-scale models without exposing sensitive personal health information.Unlike centralized AI learning,where the centralized learning machine has access to the entire training data,the federated learning process iteratively updates models across multiple sites by exchanging only parameter updates,not personal health data.This narrative review covers the basic concepts,highlights relevant corner-stone and stateof-the-art research in the field,and discusses perspectives.

An Intelligent Algorithm for Solving Weapon-Target Assignment Problem:DDPG-DNPE Algorithm

Aiming at the problems of traditional dynamic weapon-target assignment algorithms in command decisionmaking,such as large computational amount,slow solution speed,and low calculation accuracy,combined with deep reinforcement learning theory,an improved Deep Deterministic Policy Gradient algorithm with dual noise and prioritized experience replay is proposed,which uses a double noise mechanism to expand the search range of the action,and introduces a priority experience playback mechanism to effectively achieve data utilization.Finally,the algorithm is simulated and validated on the ground-to-air countermeasures digital battlefield.The results of the experiment show that,under the framework of the deep neural network for intelligent weapon-target assignment proposed in this paper,compared to the traditional RELU algorithm,the agent trained with reinforcement learning algorithms,such asDeepDeterministic Policy Gradient algorithm,Asynchronous Advantage Actor-Critic algorithm,Deep Q Network algorithm performs better.It shows that the use of deep reinforcement learning algorithms to solve the weapon-target assignment problem in the field of air defense operations is scientific.In contrast to other reinforcement learning algorithms,the agent trained by the improved Deep Deterministic Policy Gradient algorithm has a higher win rate and reward in confrontation,and the use of weapon resources is more efficient.It shows that the model and algorithm have certain superiority and rationality.The results of this paper provide new ideas for solving the problemof weapon-target assignment in air defense combat command decisions.

Thermoelectric energy harvesting for internet of things devices using machine learning:A review

Initiatives to minimise battery use,address sustainability,and reduce regular maintenance have driven the challenge to use alternative power sources to supply energy to devices deployed in Internet of Things(IoT)networks.As a key pillar of fifth generation(5G)and beyond 5G networks,IoT is estimated to reach 42 billion devices by the year 2025.Thermoelectric generators(TEGs)are solid state energy harvesters which reliably and renewably convert thermal energy into electrical energy.These devices are able to recover lost thermal energy,produce energy in extreme environments,generate electric power in remote areas,and power micro‐sensors.Applying the state of the art,the authorspresent a comprehensive review of machine learning(ML)approaches applied in combination with TEG‐powered IoT devices to manage and predict available energy.The application areas of TEG‐driven IoT devices that exploit as a heat source the temperature differences found in the environment,biological structures,machines,and other technologies are summarised.Based on detailed research of the state of the art in TEG‐powered devices,the authors investigated the research challenges,applied algorithms and application areas of this technology.The aims of the research were to devise new energy prediction and energy management systems based on ML methods,create supervised algorithms which better estimate incoming energy,and develop unsupervised and semi‐supervised ap-proaches which provide adaptive and dynamic operation.The review results indicate that TEGs are a suitable energy harvesting technology for low‐power applications through their scalability,usability in ubiquitous temperature difference scenarios,and long oper-ating lifetime.However,TEGs also have low energy efficiency(around 10%)and require a relatively constant heat source.

Impact of Artificial Intelligence on Corporate Leadership

Artificial Intelligence (AI) is transforming organizational dynamics, and revolutionizing corporate leadership practices. This research paper delves into the question of how AI influences corporate leadership, examining both its advantages and disadvantages. Positive impacts of AI are evident in communication, feedback systems, tracking mechanisms, and decision-making processes within organizations. AI-powered communication tools, as exemplified by Slack, facilitate seamless collaboration, transcending geographical barriers. Feedback systems, like Adobe’s Performance Management System, employ AI algorithms to provide personalized development opportunities, enhancing employee growth. AI-based tracking systems optimize resource allocation, as exemplified by studies like “AI-Based Tracking Systems: Enhancing Efficiency and Accountability.” Additionally, AI-powered decision support, demonstrated during the COVID-19 pandemic, showcases the capability to navigate complex challenges and maintain resilience. However, AI adoption poses challenges in human resources, potentially leading to job displacement and necessitating upskilling efforts. Managing AI errors becomes crucial, as illustrated by instances like Amazon’s biased recruiting tool. Data privacy concerns also arise, emphasizing the need for robust security measures. The proposed solution suggests leveraging Local Machine Learning Models (LLMs) to address data privacy issues. Approaches such as federated learning, on-device learning, differential privacy, and homomorphic encryption offer promising strategies. By exploring the evolving dynamics of AI and leadership, this research advocates for responsible AI adoption and proposes LLMs as a potential solution, fostering a balanced integration of AI benefits while mitigating associated risks in corporate settings.

实现E-learning平台中的学生自适应学习

国内学生长期依赖面授教学,造成学习主动性差,难以适应e-learning。本文试图以清华大学远程教育的网络教学平台为例,探索智能评价系统的设计与实现,寻求在网络教学环境下激励和引导学习的有效方法。

The pioneer of intelligent construction—An overview of the development of intelligent compaction

As the pioneer in the intelligent construction technologies(ICT)of transportation infrastructure,intelligent compaction(IC)has been applied in the infrastructure construction of various countries.It is currently the technology that best reflects the intelligence of engineering construction.This article overviews the latest developments and trends in IC.Firstly,the basic meaning of ICT is defined based on the essential characteristics of intelligent construction of transportation infrastructure,“perception,analysis,decision-making,execution”(PADE).The concept of intelligent compaction technology classification is also introduced.The PADE requirements that intelligent compaction should meet are proposed.Secondly,according to the sequence of“perception,analysis,decision-making,execution,”the workflow and key technologies of intelligent compaction are analyzed,and the mechanism of using the response of the roller to solve the modulus is given and verified.On this basis,The IC feasibility test methods,including compaction degree,compaction stability,and compaction uniformity,are briefly described.The implementation scheme of feedback control is given.Then,the use of artificial neural networks(ANN),hybrid expert systems,and reinforcement learning in intelligent compaction are briefly introduced.Finally,several extended applications of intelligent compaction are expounded,including the development ideas of intelligent road rollers and the role of intelligent compaction in virtual construction,the layer-specific mechanical parameters of fillers,etc.

A Framework for Active Learning of Beam Alignment in Vehicular Millimeter Wave Communications by Onboard Sensors

Estimating time-selective millimeter wave wireless channels and then deriving the optimum beam alignment for directional antennas is a challenging task.To solve this problem,one can focus on tracking the strongest multipath components(MPCs).Aligning antenna beams with the tracked MPCs increases the channel coherence time by several orders of magnitude.This contribution suggests tracking the MPCs geometrically.The derived geometric tracker is based on algorithms known as Doppler bearing tracking.A recent work on geometric-polar tracking is reformulated into an efficient recursive version.If the relative position of the MPCs is known,all other sensors on board a vehicle,e.g.,lidar,radar,and camera,will perform active learning based on their own observed data.By learning the relationship between sensor data and MPCs,onboard sensors can participate in channel tracking.Joint tracking of many integrated sensors will increase the reliability of MPC tracking.

基于数字孪生和深度强化学习的矿井超前液压支架自适应抗冲支护方法

受深部开采冲击地压等地质灾害扰动的影响,存在矿井超前支护系统自感知能力差、智能抗冲自适应能力弱、缺乏决策控制能力等问题。针对上述问题,提出了一种基于数字孪生和深度强化学习的矿井超前液压支架自适应抗冲支护方法。通过多源传感器感知巷道环境和超前液压支架支护状态,在虚拟世界中创建物理实体的数字孪生模型,其中物理模型精确展现超前液压支架的结构特征和细节,控制模型实现超前液压支架的自适应控制,机理模型实现对超前液压支架自适应支护的逻辑描述和机理解释,数据模型存储超前液压支架实体运行数据和孪生数据,仿真模型完成超前液压支架立柱仿真以实现超前液压支架与数字孪生模型虚实交互。根据基于深度Q网络(DQN)的超前液压支架自适应抗冲决策算法,对仿真环境中巷道抗冲支护进行智能决策,并依据决策结果对物理实体和数字孪生模型下达调控指令,实现超前液压支架智能控制。实验结果表明:立柱位移与压力变化一致,说明超前液压支架立柱仿真模型设计合理,从而验证了数字孪生模型的准确性;基于DQN的矿井超前液压支架自适应抗冲决策算法可通过调节液压支架控制器PID参数,自适应调控立柱压力,提升巷道安全等级,实现超前液压支架自适应抗冲支护。

基于自注意力机制的深度强化学习交通信号控制

交通信号控制(Traffic Signal Control, TSC)仍然是交通领域中最重要的研究课题之一。针对现有基于深度强化学习(Deep Reinforcement Learning, DRL)的交通信号控制方法的状态需要人为设计,导致提取交通状态信息难度大以及交通状态信息无法全面表达的问题,为了从有限特征中挖掘潜在交通状态信息,从而降低交通状态设计难度,提出一种引入自注意力网络的DRL算法。首先,仅获取交叉口各进入口车道车辆位置,使用非均匀量化和独热编码方法预处理得到车辆位置分布矩阵;其次,使用自注意力网络挖掘车辆位置分布矩阵的空间相关性和潜在信息,作为DRL算法的输入;最后,在单交叉口学习交通信号自适应控制策略,在多交叉口路网中验证所提算法的适应性和鲁棒性。仿真结果表明,在单交叉口环境下,与3种基准算法相比,所提算法在车辆平均等待时间等指标上具有更好的性能;在多交叉口路网中,所提算法仍然具有良好的适应性。

智能技术赋能虚拟科学探究学习过程评价与适应性反馈研究

虚拟科学探究学习过程的自动评价和适应性反馈是提升探究学习效果的重要支撑手段。研究首先从虚拟科学探究学习过程要素表征和分析模型、自动分析与实时评价、适应性反馈三个方面概述了智能技术在虚拟科学探究学习中的应用现状,总结了当前技术应用面临的深层次探究要素表征难、不确定探究过程刻画难、适应性反馈生成难等现实挑战。其次,在此基础上,提出了基于活动流的底层计算模型构建、复杂探究过程动态监测与自动评价、可解释性归因的自适应反馈内容生成三个关键技术。再次,基于关键技术设计了虚拟科学实验自主探究学习平台的技术架构。最后,研究总结了当前研究的创新之处并提出未来建议,为智能技术赋能虚拟科学探究学习领域开展更深入的技术探索提供有益参考。

大模型时代的智适应学习研究:进展、实例与展望

智适应系统和生成式人工智能的有机结合正在重塑教育,这是经济全球化、技术变革的新诉求,也是人类文明进步发展的必然阶段。在回顾智适应学习的历史发展和技术优势的基础上,探讨全球视野下智适应学习的研究进展,明晰智适应学习的五个层级,依次为互联网教育、智能工具、智适应学习、高级智适应学习、完全智适应学习。在技术层面,重点介绍混合专家教育大模型的核心开发模块,讨论如何构建大模型与智适应知识图谱、推荐系统之间的相互赋能,有机结合,最终形成以大模型为核心的人工智能体。同时依托智适应学习应用的技术架构,呈现其应用的典型案例。在此基础上,围绕智适应学习的技术应用探索,梳理和归纳智适应学习的未来愿景:一是构建全民科学教育的标准;二是创造人工智能与人类智慧的同步进化;三是通过高质量人才培养服务教育强国建设。

行为认知视域下智适应融合式学习对学生的影响探究

随着人工智能教育的发展,智适应学习系统逐步得到应用。为帮助学生提升学习效果,以行为认知视角切入,对不同学习方式的学生进行问卷调查,针对调查结果分别使用层次综合评价模型研究智适应融合式学习与传统学习方式对学生学习成效的差异性,并使用结构方程模型研究智适应融合式学习方式对学生影响的具体路径及影响程度,从而得出相关研究结论,并对未来教育学习方式提出建议与展望。

基于集成学习的交通事故严重程度预测研究与应用

目前自动驾驶技术重点是关注如何主动避免碰撞,然而在面对其他交通参与者入侵而导致不可避免的碰撞事故场景时,预测车辆在不同行驶模式下的碰撞严重程度来降低事故严重程度的研究却很少。为此,提出一种双层Stacking事故严重程度预测模型。基于真实交通事故数据集NASS-CDS完成训练,模型输入为车辆传感器可感知得到的事故相关特征,输出为车内乘员最高受伤级别。在第1层中,通过实验对不同学习器组合进行训练,最终综合考虑预测性能以及耗时挑选K近邻、自适应提升树、极度梯度提升树作为基学习器;在第2层中,为降低过拟合,采用逻辑回归作为元学习器。实验结果表明,该方法准确率达到85.01%,在精确率、召回率和F1值方面优于其他个体模型和集成模型,该预测结果可作为智能车辆决策规划模块先验信息,帮助车辆做出正确的决策,减缓事故损害。最后阐述了模型在L_(2)辅助驾驶与L_(4)自动驾驶车辆中的应用,在常规车辆安全防护的基础上进一步提升车辆的安全性。