结合APF和改进DDQN的动态环境机器人路径规划方法

针对动态环境的机器人路径规划,人工势场法(APF)易陷入局部最小陷阱;强化学习深度双重Q网络(DDQN)算法存在盲目探索过多、收敛较慢和规划路径不平滑的问题,本文提出一种基于人工势场法和改进DDQN的动态环境机器人路径规划算法(PF-IDDQN).首先,将人工势场法引入改进DDQN以获取初始全局环境信息,并对奖励模块进行优化;其次,在算法状态集中增加4个方向因素,以提高规划路径的平滑度;最后,进行了动态环境下的训练仿真.结果表明,机器人在动态环境中可以在有限探索次数内到达目标位置,验证了本文算法的有效性.

基于DDQN算法的混流车间作业动态自适应调度的研究

大规模生产的混流车间制造系统存在资源规模大、约束多等问题,快速找到合适的调度策略是实现高效生产的关键。为解决传统数学规划算法和启发式算法存在的策略求解效率低、自适应性差等问题,文章提出一种基于DDQN的智能车间动态自适应调度方法,对车间作业的自适应调度做了研究。通过“一步一推理”的自适用动态调度,可以高效地匹配合适的调度策略动作。

B-Spline-Based Curve Fitting to Cam Pitch Curve Using Reinforcement Learning

Directly applying the B-spline interpolation function to process plate cams in a computer numerical control(CNC)system may produce verbose tool-path codes and unsmooth trajectories.This paper is devoted to addressing the problem of B-splinefitting for cam pitch curves.Considering that the B-spline curve needs to meet the motion law of the follower to approximate the pitch curve,we use the radial error to quantify the effects of thefitting B-spline curve and the pitch curve.The problem thus boils down to solving a difficult global optimization problem tofind the numbers and positions of the control points or data points of the B-spline curve such that the cumulative radial error between thefitting curve and the original curve is minimized,and this problem is attempted in this paper with a double deep Q-network(DDQN)reinforcement learning(RL)algorithm with data points traceability.Specifically,the RL envir-onment,actions set and current states set are designed to facilitate the search of the data points,along with the design of the reward function and the initialization of the neural network.The experimental results show that when the angle division value of the actions set isfixed,the proposed algorithm can maximize the number of data points of the B-spline curve,and accurately place these data points to the right positions,with the minimum average of radial errors.Our work establishes the theoretical foundation for studying splinefitting using the RL method.

基于深度双Q网络的权值时变路网路径规划

针对传统路径规划方法无法根据城市路网权值时变特性规划最优路径的问题,提出了基于深度双Q网络的权值时变路网路径规划方法。首先,构建权值时变的城市路网模型,其中,路段各时间段权值由随机函数产生。然后,设计了状态特征、交互动作和奖励函数对权值时变路网路径规划问题进行建模,利用DDQN算法训练智能体来学习路网权值时变特性,最后根据建模后的状态特征实现权值时变路网的有效路径规划。实验结果表明,DDQN算法训练的智能体在权值时变路网中具有较好全局寻优能力。相比于滚动路径规划算法,所提方法在不同情况下规划的路径均最优,为权值时变路网的路径规划提供了一种新思路。

Deep reinforcement learning for UAV swarm rendezvous behavior

The unmanned aerial vehicle(UAV)swarm technology is one of the research hotspots in recent years.With the continuous improvement of autonomous intelligence of UAV,the swarm technology of UAV will become one of the main trends of UAV development in the future.This paper studies the behavior decision-making process of UAV swarm rendezvous task based on the double deep Q network(DDQN)algorithm.We design a guided reward function to effectively solve the problem of algorithm convergence caused by the sparse return problem in deep reinforcement learning(DRL)for the long period task.We also propose the concept of temporary storage area,optimizing the memory playback unit of the traditional DDQN algorithm,improving the convergence speed of the algorithm,and speeding up the training process of the algorithm.Different from traditional task environment,this paper establishes a continuous state-space task environment model to improve the authentication process of UAV task environment.Based on the DDQN algorithm,the collaborative tasks of UAV swarm in different task scenarios are trained.The experimental results validate that the DDQN algorithm is efficient in terms of training UAV swarm to complete the given collaborative tasks while meeting the requirements of UAV swarm for centralization and autonomy,and improving the intelligence of UAV swarm collaborative task execution.The simulation results show that after training,the proposed UAV swarm can carry out the rendezvous task well,and the success rate of the mission reaches 90%.

基于深度强化学习的Boost变换器控制策略

针对Boost变换器母线电压稳定控制存在模型不确定性和非线性的问题,提出了一种基于无模型深度强化学习的智能控制策略。结合强化学习DDQN(double DQN)算法与DDPG算法设计了Boost变换器控制器,包括了状态、动作空间、奖励函数以及神经网络的设计以提高控制器动态性能;基于ModelicaGym库开发工具包reinforment learning modelica(RLM)实现了Boost变换器模型与强化学习智能体的联合仿真。通过与双环PI控制器的对比仿真表明:强化学习控制器在三种工况下的母线电压稳定控制结果具有更好的动态性能。

基于深度强化学习的平滑车速控制系统

为调整不同路段的限速值,平滑交通流,从而提升高速公路车辆通行的安全性和效率,针对交通瓶颈区设计一种基于深度强化学习的平滑车速管控系统。该系统主要包含动态限速启动、限速值确定与更新和情报板动态发布等3个模块。将深度强化学习算法DDQN(Double Deep Q-Network)引入系统中,提出一种基于DDQN的平滑车速控制策略,从目标网络和经验回顾2个维度提升该算法的性能。基于元胞传输模型(Cellular Transmission Model,CTM)对宁夏高速公路某路段的交通流运行场景进行仿真,以车辆总通行时间和车流量为评价指标验证该系统的有效性,结果表明该系统能提高瓶颈区内拥堵路段车辆的通行效率。

基于Double Deep Q Network的无人机隐蔽接敌策略

基于深度强化学习的连续状态空间无人机隐蔽接敌问题,提出了基于马尔可夫决策过程的隐蔽接敌双深度Q网络(DDQN)方法。利用DDQN生成目标值函数的方法解决了传统DQN的过拟合问题;采用按优先级随机抽样的方法获取训练样本,加速了神经网络的训练速度;设定贪婪系数按照指数下降的方法,解决了传统强化学习的“探索利用窘境”;在势函数奖赏函数设计中引入角度因子,使其更加符合实际作战情况。仿真实验结果表明,DDQN具有较好的收敛性,能有效生成隐蔽接敌策略。

基于卡尔曼滤波和DDQN算法的无人机机动目标跟踪

为使无人机能够自主、准确地预测目标状态,进而对敌方机动目标进行跟踪,基于卡尔曼滤波和深度强化学习DDQN算法提出了一种在线决策算法。通过构建无人机机动目标跟踪模型和马尔科夫决策过程框架,结合卡尔曼滤波对目标状态进行了准确预测和更新;然后整合无人机自身状态作为神经网络输入,利用DDQN算法进行针对性训练,实现了无人机对机动目标的自主跟踪控制。仿真实验证明,相较于经典DQN算法,基于DDQN算法训练后的无人机,在跟踪任务中能够对目标保持更长的有效跟踪时间、跟踪距离更近,并保持更稳定的飞行状态,最终实现对机动目标的高效跟踪。