Decision-Making Models Based on Meta-Reinforcement Learning for Intelligent Vehicles at Urban Intersections

Behavioral decision-making at urban intersections is one of the primary difficulties currently impeding the development of intelligent vehicle technology.The problem is that existing decision-making algorithms cannot effectively deal with complex random scenarios at urban intersections.To deal with this,a deep deterministic policy gradient(DDPG)decision-making algorithm(T-DDPG)based on a time-series Markov decision process(T-MDP)was developed,where the state was extended to collect observations from several consecutive frames.Experiments found that T-DDPG performed better in terms of convergence and generalizability in complex intersection scenarios than a traditional DDPG algorithm.Furthermore,model-agnostic meta-learning(MAML)was incorporated into the T-DDPG algorithm to improve the training method,leading to a decision algorithm(T-MAML-DDPG)based on a secondary gradient.Simulation experiments of intersection scenarios were carried out on the Gym-Carla platform to verify and compare the decision models.The results showed that T-MAML-DDPG was able to easily deal with the random states of complex intersection scenarios,which could improve traffic safety and efficiency.The above decision-making models based on meta-reinforcement learning are significant for enhancing the decision-making ability of intelligent vehicles at urban intersections.

Enhanced UAV Pursuit-Evasion Using Boids Modelling:A Synergistic Integration of Bird Swarm Intelligence and DRL

TheUAV pursuit-evasion problem focuses on the efficient tracking and capture of evading targets using unmanned aerial vehicles(UAVs),which is pivotal in public safety applications,particularly in scenarios involving intrusion monitoring and interception.To address the challenges of data acquisition,real-world deployment,and the limited intelligence of existing algorithms in UAV pursuit-evasion tasks,we propose an innovative swarm intelligencebased UAV pursuit-evasion control framework,namely“Boids Model-based DRL Approach for Pursuit and Escape”(Boids-PE),which synergizes the strengths of swarm intelligence from bio-inspired algorithms and deep reinforcement learning(DRL).The Boids model,which simulates collective behavior through three fundamental rules,separation,alignment,and cohesion,is adopted in our work.By integrating Boids model with the Apollonian Circles algorithm,significant improvements are achieved in capturing UAVs against simple evasion strategies.To further enhance decision-making precision,we incorporate a DRL algorithm to facilitate more accurate strategic planning.We also leverage self-play training to continuously optimize the performance of pursuit UAVs.During experimental evaluation,we meticulously designed both one-on-one and multi-to-one pursuit-evasion scenarios,customizing the state space,action space,and reward function models for each scenario.Extensive simulations,supported by the PyBullet physics engine,validate the effectiveness of our proposed method.The overall results demonstrate that Boids-PE significantly enhance the efficiency and reliability of UAV pursuit-evasion tasks,providing a practical and robust solution for the real-world application of UAV pursuit-evasion missions.

Research on active defense decision-making method for cloud boundary networks based on reinforcement learning of intelligent agent

The cloud boundary network environment is characterized by a passive defense strategy,discrete defense actions,and delayed defense feedback in the face of network attacks,ignoring the influence of the external environment on defense decisions,thus resulting in poor defense effectiveness.Therefore,this paper proposes a cloud boundary network active defense model and decision method based on the reinforcement learning of intelligent agent,designs the network structure of the intelligent agent attack and defense game,and depicts the attack and defense game process of cloud boundary network;constructs the observation space and action space of reinforcement learning of intelligent agent in the non-complete information environment,and portrays the interaction process between intelligent agent and environment;establishes the reward mechanism based on the attack and defense gain,and encourage intelligent agents to learn more effective defense strategies.the designed active defense decision intelligent agent based on deep reinforcement learning can solve the problems of border dynamics,interaction lag,and control dispersion in the defense decision process of cloud boundary networks,and improve the autonomy and continuity of defense decisions.

A Machine-Learning Based Time Constrained Resource Allocation Scheme for Vehicular Fog Computing

Through integrating advanced communication and data processing technologies into smart vehicles and roadside infrastructures,the Intelligent Transportation System(ITS)has evolved as a promising paradigm for improving safety,efficiency of the transportation system.However,the strict delay requirement of the safety-related applications is still a great challenge for the ITS,especially in dense traffic environment.In this paper,we introduce the metric called Perception-Reaction Time(PRT),which reflects the time consumption of safety-related applications and is closely related to road efficiency and security.With the integration of the incorporating information-centric networking technology and the fog virtualization approach,we propose a novel fog resource scheduling mechanism to minimize the PRT.Furthermore,we adopt a deep reinforcement learning approach to design an on-line optimal resource allocation scheme.Numerical results demonstrate that our proposed schemes is able to reduce about 70%of the RPT compared with the traditional approach.

MADDPG-D2: An Intelligent Dynamic Task Allocation Algorithm Based on Multi-Agent Architecture Driven by Prior Knowledge

Aiming at the problems of low solution accuracy and high decision pressure when facing large-scale dynamic task allocation(DTA)and high-dimensional decision space with single agent,this paper combines the deep reinforce-ment learning(DRL)theory and an improved Multi-Agent Deep Deterministic Policy Gradient(MADDPG-D2)algorithm with a dual experience replay pool and a dual noise based on multi-agent architecture is proposed to improve the efficiency of DTA.The algorithm is based on the traditional Multi-Agent Deep Deterministic Policy Gradient(MADDPG)algorithm,and considers the introduction of a double noise mechanism to increase the action exploration space in the early stage of the algorithm,and the introduction of a double experience pool to improve the data utilization rate;at the same time,in order to accelerate the training speed and efficiency of the agents,and to solve the cold-start problem of the training,the a priori knowledge technology is applied to the training of the algorithm.Finally,the MADDPG-D2 algorithm is compared and analyzed based on the digital battlefield of ground and air confrontation.The experimental results show that the agents trained by the MADDPG-D2 algorithm have higher win rates and average rewards,can utilize the resources more reasonably,and better solve the problem of the traditional single agent algorithms facing the difficulty of solving the problem in the high-dimensional decision space.The MADDPG-D2 algorithm based on multi-agent architecture proposed in this paper has certain superiority and rationality in DTA.

The pioneer of intelligent construction—An overview of the development of intelligent compaction

As the pioneer in the intelligent construction technologies(ICT)of transportation infrastructure,intelligent compaction(IC)has been applied in the infrastructure construction of various countries.It is currently the technology that best reflects the intelligence of engineering construction.This article overviews the latest developments and trends in IC.Firstly,the basic meaning of ICT is defined based on the essential characteristics of intelligent construction of transportation infrastructure,“perception,analysis,decision-making,execution”(PADE).The concept of intelligent compaction technology classification is also introduced.The PADE requirements that intelligent compaction should meet are proposed.Secondly,according to the sequence of“perception,analysis,decision-making,execution,”the workflow and key technologies of intelligent compaction are analyzed,and the mechanism of using the response of the roller to solve the modulus is given and verified.On this basis,The IC feasibility test methods,including compaction degree,compaction stability,and compaction uniformity,are briefly described.The implementation scheme of feedback control is given.Then,the use of artificial neural networks(ANN),hybrid expert systems,and reinforcement learning in intelligent compaction are briefly introduced.Finally,several extended applications of intelligent compaction are expounded,including the development ideas of intelligent road rollers and the role of intelligent compaction in virtual construction,the layer-specific mechanical parameters of fillers,etc.

An Intelligent Algorithm for Solving Weapon-Target Assignment Problem:DDPG-DNPE Algorithm

Aiming at the problems of traditional dynamic weapon-target assignment algorithms in command decisionmaking,such as large computational amount,slow solution speed,and low calculation accuracy,combined with deep reinforcement learning theory,an improved Deep Deterministic Policy Gradient algorithm with dual noise and prioritized experience replay is proposed,which uses a double noise mechanism to expand the search range of the action,and introduces a priority experience playback mechanism to effectively achieve data utilization.Finally,the algorithm is simulated and validated on the ground-to-air countermeasures digital battlefield.The results of the experiment show that,under the framework of the deep neural network for intelligent weapon-target assignment proposed in this paper,compared to the traditional RELU algorithm,the agent trained with reinforcement learning algorithms,such asDeepDeterministic Policy Gradient algorithm,Asynchronous Advantage Actor-Critic algorithm,Deep Q Network algorithm performs better.It shows that the use of deep reinforcement learning algorithms to solve the weapon-target assignment problem in the field of air defense operations is scientific.In contrast to other reinforcement learning algorithms,the agent trained by the improved Deep Deterministic Policy Gradient algorithm has a higher win rate and reward in confrontation,and the use of weapon resources is more efficient.It shows that the model and algorithm have certain superiority and rationality.The results of this paper provide new ideas for solving the problemof weapon-target assignment in air defense combat command decisions.

一种基于强化学习的自适应多邻域人工蜂群算法

邻域拓扑是提高人工蜂群算法性能的一种有效手段.然而,现有相关工作主要是在种群层次上实现了单一邻域拓扑,这种方式忽略了不同类型的邻域拓扑能优势互补,使得算法性能还有一定局限性.为此,本文结合强化学习,提出在个体层次上实现多邻域拓扑。将种群中的个体视作智能体,设计了基于邻域拓扑的状态和动作,选用4种不同特征的邻域拓扑用于构建邻域候选池,之后采用Q-learning方法根据个体的奖励情况为其自适应选择不同的邻域拓扑.该方式相对于现有的单一邻域拓扑,更能充分发挥不同邻域信息对算法搜索的引导作用.在CEC2013和CEC2017两套测试集以及两个实际优化问题上进行了大量实验,与4种邻域相关ABC和4种知名改进ABC进行了性能对比,结果表明该算法的收敛精度和速度均有更好表现,可有效增强邻域人工蜂群算法的性能.

无人集群系统深度强化学习控制研究进展

随着无人集群在物流运输、农业管理、军事行动等场景的试验和应用,其面临的作业环境和任务内容日趋复杂,亟需设计效率更高、泛化能力更强、适应性更好的控制算法.将人工智能引入到无人集群系统控制的研究中,能够大幅提升现有无人集群的能力,完成复杂的作业任务.深度强化学习具有深度学习和强化学习的优点,无人集群系统深度强化学习控制研究受到了国内外科研人员的广泛关注,涌现出许多标志性成果.本文将从原理、特点等方面阐述深度强化学习概念,深入分析深度强化学习的多种典型算法,并讨论无人机集群的各类控制需求,进而介绍深度强化学习在无人机集群控制领域的典型研究成果,最后针对该领域研究成果的落地转化总结了应用前景和面临的挑战.

基于强化学习的无人机智能组网技术及应用综述

针对无人机在民用和军事等领域中的研究热度及应用需求日益增长,传统Mode1-Based的网络部署、设计、操作方法无法应对动态变化的无人机场景的问题,本文综述了灵活性高、适应性强的AI-Based的智能组网技术,并引入强化学习这一人工智能领域的重要分支。对现有利用强化学习技术解决无人机组网难题的研究进行了概述,结合无人机组网的特性梳理了此领域应用强化学习技术的主要思路。从几个应用场景,以及组网关键技术的角度进行了归纳,给出了基于强化学习的无人机智能组网技术所面临的机遇与挑战,并进行了总结。探究了无人机通信的感知能力与决策能力,适应了其动态变化且需要高度自治的环境需求。为未来无人机智能组网技术的发展提供了有价值的理论基础和实践指导。

智能网联汽车中联合驾驶风格的交通流数据有效性分析

在智能网联汽车(ICV)中,提高驾驶数据的有效性是提升车辆安全性的基石。只有准确的、可靠的驾驶数据才能为车辆的安全性提供可靠的依据和支持。与传统的异常分析相比,ICV数据有效性分析面临数据异常的多样性(传感器异常、驾驶行为、恶意篡改等)。如何将车辆自身数据特征、驾驶风格和交通流特征相结合,提供有效的数据异常检测方法,已成为智能网联汽车中新的问题。针对ICV系统,采用结合驾驶风格和交通流理论的方法,设计基于粒子群优化的TE-PSO-SVM数据有效性检测算法,实现驾驶数据的有效检测。首先定义驾驶风格识别系数Rad,设计驾驶风格量化模型;其次建立交通流模型,将车辆状态数据与驾驶风格和交通流理论相结合,通过长短期记忆(LSTM)网络对车辆速度预测;最后通过TE-PSO-SVM算法进行数据有效性检测。由于ICV数据的多样性,单一模型在多类型异常混合并存的场景中检测精度仍有局限,利用多个模型的优势构建模型池,并提出基于强化学习的模型选择(RLBMS)算法。在真实数据集highD上的实验结果表明,在不同噪声环境下,TE-PSO-SVM算法模型的F1值相比于传统SVM模型平均提升约8.1个百分点;RLBMS算法模型在不同噪声环境下的F1值相比于模型池中检测率最高的算法平均提高约1.7个百分点,明显提高了数据有效性检测的准确率。

群体追逃微分博弈

本文以微分博弈和经典的追逃问题为主线,对群体追逃微分博弈的历史发展脉络进行梳理。针对大规模群体追逃问题,从平均场博弈视角出发,阐释了强化学习技术的应用前景。提出探索解决逆向追逃微分博弈的观点,可适用于水下无人舰艇、陆地机器人以及空中无人机集群等同类场景。区别于其他综述性文章,作者对于俄罗斯以及苏联在本领域发展历史中代表性的学术流派给予了较多关注。

基于多智能体深度强化学习的无人艇集群博弈对抗研究

基于未来现代化海上作战背景,提出了利用多智能体深度强化学习方案来完成无人艇群博弈对抗中的协同围捕任务。首先,根据不同的作战模式和应用场景,提出基于分布式执行的多智能体深度确定性策略梯度算法,并对其原理进行了介绍;其次,模拟具体作战场景平台,设计多智能体网络模型、奖励函数机制以及训练策略。实验结果表明,文中方法可以有效应对敌方无人艇的协同围捕决策问题,在不同作战场景下具有较高的效率,为未来复杂作战场景下无人艇智能决策研究提供理论参考价值。

基于学习型算法的农资配送优化问题研究

针对农资供应链订单配送路径优化问题,考虑新能源货车综合续航里程、车辆最大载荷能力和时间窗等约束,建立车辆路径规划问题的数学模型(DCVRPTW)综合优化车辆固定成本和运输成本,提出一种基于深度强化学习的群智能优化算法框架(DRL-SIA)。智能体就是决策者,以环境状态为输入选出动作池中最佳动作改变环境并获得环境奖励。DRL-SIA算法结合训练后的智能体与群智能算法以代替原算法进行决策选择,从而提升寻优速度与精度。实验表明,所提算法的最优解相较于其他算法在所有算例中最优,验证了该算法能有效降低农用物资供应链中的物流运输成本。

Brain-like Intelligent Decision-making Based on Basal Ganglia and Its Application in Automatic Car-following

The anthropomorphic intelligence of autonomous driving has been a research hotspot in the world.However,current studies have not been able to reveal the mechanism of drivers'natural driving behaviors.Therefore,this thesis starts from the perspective of cognitive decision-making in the human brain,which is inspired by the regulation of dopamine feedback in the basal ganglia,and a reinforcement learning model is established to solve the brain-like intelligent decision-making problems in the process of interacting with the environment.In this thesis,first,a detailed bionic mechanism architecture based on basal ganglia was proposed by the consideration and analysis of its feedback regulation mechanism;second,the above mechanism was transformed into a reinforcement Q-learning model,so as to implement the learning and adaptation abilities of an intelligent vehicle for brain-like intelligent decision-making during car-following;finally,the feasibility and effectiveness of the proposed method were verified by the simulations and real vehicle tests.

面向复杂环境的集群无人机任务调度方法研究综述

近年来,无人机由于其成本低、速度快和灵活性强等优点,在军事和民用领域得到了广泛应用。集群无人机是由一组同构或异构无人机组成,通过个体自主决策和信息交互,实现感知互动、信息传递和协同工作。相较于单一无人机,集群无人机可利用其集群优势、自主优势及智能化优势完成复杂任务。然而,随着任务环境、需求和集群规模的不断变化,集群无人机任务调度问题成为备受关注的热点问题。总结近几年的代表性研究,列举了复杂环境下集群无人机任务调度面临的挑战:动态任务需求、复杂环境条件、不确定通信条件及资源受限。随后,按照调度算法的作用机理划分了当前主流的调度方法,即优化算法、演化算法、强化学习算法以及群体智能算法,并对上述方法的原理、研究现状进行了归纳总结。最后,对集群无人机任务调度的未来研究方向进行了展望。

无人机群分簇路由算法研究

阐述无人机群的分簇路由算法.介绍了无人机分簇路由算法的相关概念和路由生成方法,基于群智能的分簇路由算法的相关概念和发展现状,基于强化学习的分簇路由算法的相关概念和发展现状,展望下一步研究方向.

基于深度确定性策略梯度的粒子群算法

在传统的粒子群优化算法(PSO)中,所有粒子都遵循最初设定的一些参数进行自我探索,这种方案容易导致过早成熟,且易被困于局部最优点。针对以上问题,该文提出了一种基于深度确定性策略梯度的粒子群优化算法(DDPGPSO),通过构造神经网络分别实现了动作函数和动作价值函数,且利用神经网络可以动态地生成算法运行所需要的参数,降低了人工配置算法的难度。实验表明DDPGPSO相比9种同类算法在收敛速度和寻优精度上均有较大的提升。

基于群智能强化学习的电网最优碳-能复合流算法

结合电网能流和碳排放流的传输特性,建立了电网最优碳-能复合流的数学模型,并提出了基于群智能的多步回溯Q(λ)强化学习算法,有效解决了电网碳-能复合流的动态优化问题。其中以线性加权的方式把电网网损、碳流损耗和电压稳定设计为奖励函数,通过引入粒子群的多主体计算,每个主体都有各自的Q值矩阵进行寻优迭代。IEEE118节点仿真结果表明:较传统Q(λ)算法本文所提出算法能在保证较好全局寻优能力的同时,收敛速度至少能提高10倍以上,为解决实际大规模复杂电网的碳-能复合流在线滚动优化提供了一种快速、有效的方法。

多智能体群智博弈策略轻量化问题

未来战争中智能技术的应用,无人系统如智能弹群、无人机群等多智能体将投入作战,要求智能体具有快速作战决策能力.由于无人系统的计算资源有限、内存空间小、数据传输受限,多智能体系统的自主性、协同性及群智决策等算法应实现轻量化、开销最小化.从多智能体群智决策存在的挑战出发,提出了基于深度网络的强化学习群智决策模型,讨论了其中涉及的关键技术,创新地从OODA决策循环4个关键环节提出轻量化思路.