Improved Double Deep Q Network Algorithm Based on Average Q-Value Estimation and Reward Redistribution for Robot Path Planning

By integrating deep neural networks with reinforcement learning,the Double Deep Q Network(DDQN)algorithm overcomes the limitations of Q-learning in handling continuous spaces and is widely applied in the path planning of mobile robots.However,the traditional DDQN algorithm suffers from sparse rewards and inefficient utilization of high-quality data.Targeting those problems,an improved DDQN algorithm based on average Q-value estimation and reward redistribution was proposed.First,to enhance the precision of the target Q-value,the average of multiple previously learned Q-values from the target Q network is used to replace the single Q-value from the current target Q network.Next,a reward redistribution mechanism is designed to overcome the sparse reward problem by adjusting the final reward of each action using the round reward from trajectory information.Additionally,a reward-prioritized experience selection method is introduced,which ranks experience samples according to reward values to ensure frequent utilization of high-quality data.Finally,simulation experiments are conducted to verify the effectiveness of the proposed algorithm in fixed-position scenario and random environments.The experimental results show that compared to the traditional DDQN algorithm,the proposed algorithm achieves shorter average running time,higher average return and fewer average steps.The performance of the proposed algorithm is improved by 11.43%in the fixed scenario and 8.33%in random environments.It not only plans economic and safe paths but also significantly improves efficiency and generalization in path planning,making it suitable for widespread application in autonomous navigation and industrial automation.

基于Dueling Double DQN的交通信号控制方法

为了提高交叉口通行效率缓解交通拥堵,深入挖掘交通状态信息中所包含的深层次隐含特征信息,提出了一种基于Dueling Double DQN(D3QN)的单交叉口交通信号控制方法;构建了一个基于深度强化学习Double DQN(DDQN)的交通信号控制模型,对动作-价值函数的估计值和目标值迭代运算过程进行了优化,克服基于深度强化学习DQN的交通信号控制模型存在收敛速度慢的问题;设计了一个新的Dueling Network解耦交通状态和相位动作的价值,增强Double DQN(DDQN)提取深层次特征信息的能力;基于微观仿真平台SUMO搭建了一个单交叉口模拟仿真框架和环境,开展仿真测试;仿真测试结果表明,与传统交通信号控制方法和基于深度强化学习DQN的交通信号控制方法相比,所提方法能够有效减少车辆平均等待时间、车辆平均排队长度和车辆平均停车次数,明显提升交叉口通行效率。

Double Pruning Structure Design for Deep Stochastic Configuration Networks Based on Mutual Information and Relevance

Deep stochastic configuration networks(DSCNs)produce redundant hidden nodes and connections during training,which complicates their model structures.Aiming at the above problems,this paper proposes a double pruning structure design algorithm for DSCNs based on mutual information and relevance.During the training process,the mutual information algorithm is used to calculate and sort the importance scores of the nodes in each hidden layer in a layer-by-layer manner,the node pruning rate of each layer is set according to the depth of the DSCN at the current time,the nodes that contribute little to the model are deleted,and the network-related parameters are updated.When the model completes the configuration procedure,the correlation evaluation strategy is used to sort the global connection weights and delete insignificance connections;then,the network parameters are updated after pruning is completed.The experimental results show that the proposed structure design method can effectively compress the scale of a DSCN model and improve its modeling speed;the model accuracy loss is small,and fine-tuning for accuracy restoration is not needed.The obtained DSCN model has certain application value in the field of regression analysis.

基于Deep Q Networks的机械臂推动和抓握协同控制

针对目前机械臂在复杂场景应用不足以及推动和抓握自主协同控制研究不多的现状,发挥深度Q网络(Deep Q Networks)无规则、自主学习优势,提出了一种基于Deep Q Networks的机械臂推动和抓握协同控制方法。通过2个完全卷积网络将场景信息映射至推动或抓握动作,经过马尔可夫过程,采取目光长远奖励机制,选取最佳行为函数,实现对复杂场景机械臂推动和抓握动作的自主协同控制。在仿真和真实场景实验中,该方法在复杂场景中能够通过推动和抓握自主协同操控实现对物块的快速抓取,并获得更高的动作效率和抓取成功率。

基于Deep Q Networks的交通指示灯控制方法

交通指示灯的智能控制是当前智能交通研究中的热点问题;为更加及时有效地自适应动态交通,进一步提升街道路口车流效率,提出了一种基于Deep Q Networks的道路指示灯控制方法;该方法基于道路指示灯控制问题描述,以状态、行动和奖励三要素构建道路指示灯控制的强化学习模型,提出基于Deep Q Networks的道路指示控制方法流程;为检验方法的有效性,以浙江省台州市市府大道与东环大道交叉路口交通数据在SUMO中进行方法比对与仿真实验;实验结果表明,基于Deep Q Networks的交通指示灯控制方法在交通指示等的控制与调度中具有更高的效率和自主性,更有利于改善路口车流的吞吐量,对道路路口车流的驻留时延、队列长度和等待时间等方面的优化具有更好的性能。

Artificial Potential Field Incorporated Deep-Q-Network Algorithm for Mobile Robot Path Prediction

Autonomous navigation of mobile robots is a challenging task that requires them to travel from their initial position to their destination without collision in an environment.Reinforcement Learning methods enable a state action function in mobile robots suited to their environment.During trial-and-error interaction with its surroundings,it helps a robot tofind an ideal behavior on its own.The Deep Q Network(DQN)algorithm is used in TurtleBot 3(TB3)to achieve the goal by successfully avoiding the obstacles.But it requires a large number of training iterations.This research mainly focuses on a mobility robot’s best path prediction utilizing DQN and the Artificial Potential Field(APF)algorithms.First,a TB3 Waffle Pi DQN is built and trained to reach the goal.Then the APF shortest path algorithm is incorporated into the DQN algorithm.The proposed planning approach is compared with the standard DQN method in a virtual environment based on the Robot Operation System(ROS).The results from the simulation show that the combination is effective for DQN and APF gives a better optimal path and takes less time when compared to the conventional DQN algo-rithm.The performance improvement rate of the proposed DQN+APF in comparison with DQN in terms of the number of successful targets is attained by 88%.The performance of the proposed DQN+APF in comparison with DQN in terms of average time is achieved by 0.331 s.The performance of the proposed DQN+APF in comparison with DQN average rewards in which the positive goal is attained by 85%and the negative goal is attained by-90%.

基于Double-DQN的平衡类游戏改善

现有的强化学习算法在平衡类游戏中存在过拟合与参数设置不当等问题。针对这些问题,强化学习算法应用到平衡类游戏中,不仅可以改善游戏的平衡性降低因人工测试不到位而产生的漏洞,而且可以提高游戏二次开发的可塑性。本文基于Q-learning、DeepQ-learning network、Double deep Q-learning network3种有关Q学习的强化学习算法对平衡类游戏进行实验对比,最后由测试结果得出结论:Double Deep Q-learning network强化学习算法解决了过拟合与参数设置不当等人工测试问题,更适合应用在平衡类游戏中。

Double BP Q-Learning Algorithm for Local Path Planning of Mobile Robot

Aiming at the dimension disaster problem, poor model generalization ability and deadlock problem in special obstacles environment caused by the increase of state information in the local path planning process of mobile robot, this paper proposed a Double BP Q-learning algorithm based on the fusion of Double Q-learning algorithm and BP neural network. In order to solve the dimensional disaster problem, two BP neural network fitting value functions with the same network structure were used to replace the two Q value tables in Double Q-Learning algorithm to solve the problem that the Q value table cannot store excessive state information. By adding the mechanism of priority experience replay and using the parameter transfer to initialize the model parameters in different environments, it could accelerate the convergence rate of the algorithm, improve the learning efficiency and the generalization ability of the model. By designing specific action selection strategy in special environment, the deadlock state could be avoided and the mobile robot could reach the target point. Finally, the designed Double BP Q-learning algorithm was simulated and verified, and the probability of mobile robot reaching the target point in the parameter update process was compared with the Double Q-learning algorithm under the same condition of the planned path length. The results showed that the model trained by the improved Double BP Q-learning algorithm had a higher success rate in finding the optimal or sub-optimal path in the dense discrete environment, besides, it had stronger model generalization ability, fewer redundant sections, and could reach the target point without entering the deadlock zone in the special obstacles environment.

Walking Stability Control Method for Biped Robot on Uneven Ground Based on Deep Q-Network

A gait control method for a biped robot based on the deep Q-network (DQN) algorithm is proposed to enhance the stability of walking on uneven ground. This control strategy is an intelligent learning method of posture adjustment. A robot is taken as an agent and trained to walk steadily on an uneven surface with obstacles, using a simple reward function based on forward progress. The reward-punishment (RP) mechanism of the DQN algorithm is established after obtaining the offline gait which was generated in advance foot trajectory planning. Instead of implementing a complex dynamic model, the proposed method enables the biped robot to learn to adjust its posture on the uneven ground and ensures walking stability. The performance and effectiveness of the proposed algorithm was validated in the V-REP simulation environment. The results demonstrate that the biped robot's lateral tile angle is less than 3° after implementing the proposed method and the walking stability is obviously improved.

Improved Double Deep Q Network-Based Task Scheduling Algorithm in Edge Computing for Makespan Optimization

Edge computing nodes undertake an increasing number of tasks with the rise of business density.Therefore,how to efficiently allocate large-scale and dynamic workloads to edge computing resources has become a critical challenge.This study proposes an edge task scheduling approach based on an improved Double Deep Q Network(DQN),which is adopted to separate the calculations of target Q values and the selection of the action in two networks.A new reward function is designed,and a control unit is added to the experience replay unit of the agent.The management of experience data are also modified to fully utilize its value and improve learning efficiency.Reinforcement learning agents usually learn from an ignorant state,which is inefficient.As such,this study proposes a novel particle swarm optimization algorithm with an improved fitness function,which can generate optimal solutions for task scheduling.These optimized solutions are provided for the agent to pre-train network parameters to obtain a better cognition level.The proposed algorithm is compared with six other methods in simulation experiments.Results show that the proposed algorithm outperforms other benchmark methods regarding makespan.

一种基于DQN的去中心化优先级卸载策略

边缘计算(EC)可在网络边缘为用户提供低延迟、高响应的服务。因此,资源利用率高、时延低的任务卸载策略成为研究的热门方向。但大部分现有的任务卸载研究是基于中心化的架构,通过中心化设施制定卸载策略并进行资源调度,容易受到单点故障的影响,且会产生较多的能耗和较高的时延。针对以上问题,提出一种基于深度Q网络(DQN)的去中心化优先级(DP-DQN)卸载策略。首先,设置通信矩阵模拟现实中边缘服务器有限的通信状态;其次,通过对任务设定优先级,使任务可以在不同边缘服务器之间跳转,保证各边缘服务器均可以自主制定卸载策略,完成任务卸载的去中心化;最后,根据任务的跳转次数为任务分配更多的计算资源,提高资源利用效率和优化效果。为了验证所提策略的有效性,针对不同DQN下参数的收敛性能进行了研究对比,实验结果表明,在不同测试情景下,DP-DQN的性能均优于本地算法、完全贪婪算法和多目标任务卸载算法,性能可提升约11%~19%。

基于D3QN的火力方案优选方法

针对在多类弹药协同攻击地面工事类目标任务中火力方案优选效率低的问题,提出一种基于双层决斗DQN(dueling double deep Q network,D3QN)的火力方案优选方法。该方法将打击过程建模为马尔科夫决策过程(Markov decision processes,MDP),设计其状态空间和动作空间,设计综合奖励函数激励火力方案生成策略优化,使智能体通过强化学习框架对策略进行自主训练。仿真实验结果表明,该方法对地面工事类目标的火力方案进行决策,相较于传统启发式智能算法能够获得较优的火力方案,其计算效率和结果的稳定性相较于传统深度强化学习算法具有更明显的优势。

基于集成DQN的自适应边缘缓存算法

工业应用中,动态多变的流式数据特性使强化学习算法在训练过程中很难在模型收敛性与知识遗忘之间实现很好的平衡。考虑工业现场内容请求与当前生产任务具有高度相关性,提出一种基于集成深度Q网络算法(Integrated Deep Q-Network,IDQN)的自适应缓存策略。算法在离线阶段利用不同历史任务数据,训练并保存多个历史任务模型。在线阶段每当检测到实时数据流的任务特征发生变化,则重新训练网络模型。如果实时数据流的特征隶属于历史任务,则向深度Q网络(Deep Q-Network,DQN)导入相应的历史任务模型进行网络训练。否则直接利用实时数据流训练并标记为新的任务模型。仿真实验结果表明,IDQN与参考算法相比,在内容请求流行度动态变化时能够有效减少模型收敛时间,提高缓存效率。

基于FL-MADQN算法的NR-V2X车载通信频谱资源分配

针对5G新空口-车联网(New Radio-Vehicle to Everything,NR-V2X)场景下车对基础设施(Vehicle to Infrastructure,V2I)和车对车(Vehicle to Vehicle,V2V)共享上行通信链路的频谱资源分配问题,提出了一种联邦-多智能体深度Q网络(Federated Learning-Multi-Agent Deep Q Network,FL-MADQN)算法.该分布式算法中,每个车辆用户作为一个智能体,根据获取的本地信道状态信息,以网络信道容量最佳为目标函数,采用DQN算法训练学习本地网络模型.采用联邦学习加快以及稳定各智能体网络模型训练的收敛速度,即将各智能体的本地模型上传至基站进行聚合形成全局模型,再将全局模型下发至各智能体更新本地模型.仿真结果表明:与传统分布式多智能体DQN算法相比,所提出的方案具有更快的模型收敛速度,并且当车辆用户数增大时仍然保证V2V链路的通信效率以及V2I链路的信道容量.

演化算法的DQN网络参数优化方法

为了解决DQN(Deep Q Network)在早期会出现盲目搜索、勘探利用不均并导致整个算法收敛过慢的问题,从探索前期有利于算法训练的有效信息获取与利用的角度出发,以差分演化(Differential Evolution)算法为例,提出了一种基于演化算法优化DQN网络参数以加快其收敛速度的方法(DE-DQN)。首先,将DQN的网络参数编码为演化个体;其次,分别采用“运行步长”和“平均回报”两种适应度函数评价方式;利用CartPole控制问题进行仿真对比,验证了两种评价方式的有效性。最后,实验结果表明,在智能体训练5 000代时所提出的改进算法,以“运行步长”为适应度函数时,在运行步长、平均回报和累计回报上分别提高了82.7%,18.1%和25.1%,并优于改进DQN算法;以“平均回报”为适应度函数时,在运行步长、平均回报和累计回报上分别提高了74.9%,18.5%和13.3%并优于改进DQN算法。这说明了DE-DQN算法相较于传统的DQN及其改进算法前期能获得更多有用信息,加快收敛速度。

基于Dueling-DQN的异构无线网络垂直切换算法研究

针对当前异构无线网络中切换算法考虑的服务质量指标较少,用户频繁切换愈加严重的问题,提出了基于主客观加权与改进的深度强化学习相结合的异构无线网络垂直切换方法。首先,提出了一种支持异构无线网络的软件定义网络架构;其次,提出了主客观加权相结合的属性加权算法;最后,将网络选择问题利用改进的Dueling-DQN方法解决。仿真结果表明,本文所提算法在不同用户类型网络下切换次数分别减少了11.25%,13.34%,18.76%,13.75%,吞吐量提升了16.64%。因此本文所提算法有效避免了乒乓切换,减少切换次数并且提升了吞吐量。

基于威胁机制-双重深度Q网络的多功能雷达认知干扰决策

针对传统深度Q网络(deep Q network,DQN)在雷达认知干扰决策中容易产生经验遗忘,从而重复执行错误决策的问题,本文提出了一种基于威胁机制双重深度Q网络(threat warning mechanism-double DQN,TW-DDQN)的认知干扰决策方法,该机制包含威胁网络和经验回放2种机制。为了验证算法的有效性,在考虑多功能雷达(multifunctional radar,MFR)工作状态与干扰样式之间的关联性的前提下,搭建了基于认知电子战的仿真环境,分析了雷达与干扰机之间的对抗博弈过程,并且在使用TW-DDQN进行训练的过程中,讨论了威胁半径与威胁步长参数的不同对训练过程的影响。仿真实验结果表明,干扰机通过自主学习成功与雷达进行了长时间的博弈,有80%的概率成功突防,训练效果明显优于传统DQN和优先经验回放DDQN(prioritized experience replay-DDQN,PER-DDQN)。

基于改进联邦竞争深度Q网络的多微网能量管理策略

目前,基于联邦深度强化学习的微网(MG)能量管理研究未考虑多类型能量转换与MG间电量交易的问题,同时,频繁交互模型参数导致通信时延较大。基于此,以一种包含风、光、电、气等多类型能源的MG为研究对象,构建了支持MG间电量交易和MG内能量转换的能量管理模型,提出基于正余弦算法的联邦竞争深度Q网络学习算法,并基于该算法设计了计及能量交易与转换的多MG能量管理与优化策略。仿真结果表明,所提能量管理策略在保护数据隐私的前提下,能够得到更高奖励且最大化MG经济收益,同时降低了通信时延。

基于深度Q网络的机器人路径规划研究综述

随着深度强化学习的不断发展,深度Q网络(DQN)在机器人路径规划中得到广泛关注和研究。首先,简要介绍DQN以及Nature DQN、Double DQN、Dueling DQN和D3QN等算法的基本原理和改进思想。针对算法存在的样本获取成本高和交互效率低的问题,系统梳理并总结了从奖励函数、探索能力、样本利用率等方面进行优化的研究成果和思路。最后,讨论了DQN在现代物流中进行机器人路径规划的优势,对每个场景提出了算法的优化方向,涵盖状态空间、动作空间以及奖励函数等多个关键方面。

无人驾驶中运用DQN进行障碍物分类的避障方法

安全是无人驾驶汽车需要考虑的首要因素,而避障问题是解决驾驶安全最有效的手段。基于学习的避障方法因其能够从环境中学习并直接从感知中做出决策的能力而受到研究者的关注。深度Q网络(DQN)作为一种流行的强化学习方法,在无人驾驶避障领域取得了很大的进展,但这些方法未考虑障碍物类型对避障策略的影响。基于对障碍物的准确分类提出一种Classification Security DQN(CSDQN)的车辆行驶决策框架。根据障碍物的不同类型以及环境信息给出具有更高安全性的无人驾驶决策,达到提高无人驾驶安全性的目的。首先对检测到的障碍物根据障碍物的安全性等级进行分类,然后根据不同类型障碍物提出安全评估函数,利用位置的不确定性和基于距离的安全度量来评估安全性,接着CSDQN决策框架利用障碍物类型、相对位置信息以及安全评估函数进行不断迭代优化获得最终模型。仿真结果表明,与先进的深度强化学习进行比较,在多种障碍物的情况下,采用CSDQN方法相较于DQN和SDQN方法分别提升了43.9%和4.2%的安全性,以及17.8%和3.7%的稳定性。