Hierarchical Reinforcement Learning With Automatic Sub-Goal Identification

In reinforcement learning an agent may explore ineffectively when dealing with sparse reward tasks where finding a reward point is difficult.To solve the problem,we propose an algorithm called hierarchical deep reinforcement learning with automatic sub-goal identification via computer vision(HADS)which takes advantage of hierarchical reinforcement learning to alleviate the sparse reward problem and improve efficiency of exploration by utilizing a sub-goal mechanism.HADS uses a computer vision method to identify sub-goals automatically for hierarchical deep reinforcement learning.Due to the fact that not all sub-goal points are reachable,a mechanism is proposed to remove unreachable sub-goal points so as to further improve the performance of the algorithm.HADS involves contour recognition to identify sub-goals from the state image where some salient states in the state image may be recognized as sub-goals,while those that are not will be removed based on prior knowledge.Our experiments verified the effect of the algorithm.

Hierarchical reinforcement learning guidance with threat avoidance

The guidance strategy is an extremely critical factor in determining the striking effect of the missile operation.A novel guidance law is presented by exploiting the deep reinforcement learning(DRL)with the hierarchical deep deterministic policy gradient(DDPG)algorithm.The reward functions are constructed to minimize the line-of-sight(LOS)angle rate and avoid the threat caused by the opposed obstacles.To attenuate the chattering of the acceleration,a hierarchical reinforcement learning structure and an improved reward function with action penalty are put forward.The simulation results validate that the missile under the proposed method can hit the target successfully and keep away from the threatened areas effectively.

A special hierarchical fuzzy neural-networks based reinforcement learning for multi-variables system

Proposes a reinforcement learning scheme based on a special Hierarchical Fuzzy Neural-Networks (HFNN)for solving complicated learning tasks in a continuous multi-variables environment. The output of the previous layer in the HFNN is no longer used as if-part of the next layer, but used only in then-part. Thus it can deal with the difficulty when the output of the previous layer is meaningless or its meaning is uncertain. The proposed HFNN has a minimal number of fuzzy rules and can successfully solve the problem of rules combination explosion and decrease the quantity of computation and memory requirement. In the learning process, two HFNN with the same structure perform fuzzy action composition and evaluation function approximation simultaneously where the parameters of neural-networks are tuned and updated on line by using gradient descent algorithm. The reinforcement learning method is proved to be correct and feasible by simulation of a double inverted pendulum system.

Hierarchical Reinforcement Learning Adversarial Algorithm Against Opponent with Fixed Offensive Strategy

Based on option-critic algorithm,a new adversarial algorithm named deterministic policy network with option architecture is proposed to improve agent's performance against opponent with fixed offensive algorithm.An option network is introduced in upper level design,which can generate activated signal from defensive and of-fensive strategies according to temporary situation.Then the lower level executive layer can figure out interactive action with guidance of activated signal,and the value of both activated signal and interactive action is evaluated by critic structure together.This method could release requirement of semi Markov decision process effectively and eventually simplified network structure by eliminating termination possibility layer.According to the result of experiment,it is proved that new algorithm switches strategy style between offensive and defensive ones neatly and acquires more reward from environment than classical deep deterministic policy gradient algorithm does.

Autonomous Overtaking for Intelligent Vehicles Considering Social Preference Based on Hierarchical Reinforcement Learning

As intelligent vehicles usually have complex overtaking process,a safe and efficient automated overtaking system(AOS)is vital to avoid accidents caused by wrong operation of drivers.Existing AOSs rarely consider longitudinal reactions of the overtaken vehicle(OV)during overtaking.This paper proposed a novel AOS based on hierarchical reinforcement learning,where the longitudinal reaction is given by a data-driven social preference estimation.This AOS incorporates two modules that can function in different overtaking phases.The first module based on semi-Markov decision process and motion primitives is built for motion planning and control.The second module based on Markov decision process is designed to enable vehicles to make proper decisions according to the social preference of OV.Based on realistic overtaking data,the proposed AOS and its modules are verified experimentally.The results of the tests show that the proposed AOS can realize safe and effective overtaking in scenes built by realistic data,and has the ability to flexibly adjust lateral driving behavior and lane changing position when the OVs have different social preferences.

Autonomic discovery of subgoals in hierarchical reinforcement learning

Option is a promising method to discover the hierarchical structure in reinforcement learning （RL） for learning acceleration. The key to option discovery is about how an agent can find useful subgoals autonomically among the passing trails. By analyzing the agent＇s actions in the trails, useful heuristics can be found. Not only does the agent pass subgoals more frequently, but also its effective actions are restricted in subgoals. As a consequence, the subgoals can be deemed as the most matching action-restricted states in the paths. In the grid-world environment, the concept of the unique-direction value reflecting the action-restricted property was introduced to find the most matching action-restricted states. The unique-direction-value （UDV） approach is chosen to form options offline and online autonomically. Experiments show that the approach can find subgoals correctly. Thus the Q-learning with options found on both offline and online process can accelerate learning significantly.

基于改进SNN-HRL的智能体路径规划算法

针对SNN-HRL等传统Skill discovery类算法存在的探索困难问题,本文基于SNN-HRL算法提出了融合多种探索策略的分层强化学习算法MES-HRL,改进传统分层结构,算法包括探索轨迹、学习轨迹、路径规划三层.在探索轨迹层,训练智能体尽可能多地探索未知环境,为后续的训练过程提供足够的环境状态信息.在学习轨迹层,将探索轨迹层的训练结果作为“先验知识”用于该层训练,提高训练效率.在路径规划层,利用智能体之前获得的skill来完成路径规划任务.通过仿真对比MES-HRL与SNN-HRL算法在不同环境下的性能表现,仿真结果显示,MES-HRL算法解决了传统算法的探索问题,具有更出色的路径规划能力.

基于分层约束强化学习的综合能源多微网系统优化调度

构建多微网系统是消纳可再生能源、提升电网稳定性的有效方式。通过各微网的协调调度,可有效提升微网的运行效益以及可再生能源的消纳水平。现有多微网优化问题场景多元,变量众多,再加上源荷不确定性及多微网主体的数据隐私保护等问题,为模型的高效求解带来了巨大挑战。为此,该文提出了一种分层约束强化学习优化方法。首先,构建了多微网分层强化学习优化框架,上层由智能体给出各微网储能优化策略和微网间功率交互策略;下层各微网以上层策略为约束,基于自身状态信息采用数学规划法对各微网内部的分布式电源出力进行自治优化。通过分层架构,减小通信压力,保护微网内部数据隐私,充分发挥强化学习对源荷不确定性的自适应能力,大幅提升了模型求解速度,并有效兼顾了数学规划法的求解精度。此外,将拉格朗日乘子法与传统强化学习方法相结合,提出一种约束强化学习求解方法,有效地解决了传统强化学习方法难以处理的约束越限问题。最后通过算例验证了该方法的有效性和优势。

基于分层强化学习的低过载比拦截制导律

为解决低过载比和纯角度量测等约束下的三维机动目标拦截制导问题,提出了一种基于分层强化学习的拦截制导律。首先将问题建模为马尔科夫决策过程模型,并考虑拦截能量消耗与弹目视线角速率,设计了一种启发式奖赏函数。其次通过构建具有双层结构的策略网络,并利用上层策略规划阶段性子目标来指导下层策略生成所需的制导指令,实现了拦截交战过程中的视线角速率收敛,以保证能成功拦截机动目标。仿真结果验证了所提出的方法较增强比例导引具有更高的拦截精度和拦截概率,且拦截过程的需用过载更低。

基于近似动态规划的多级火箭全程任务决策

针对火箭发生推力下降故障下的任务决策问题,提出了一种基于近似动态规划的多级火箭全程任务决策方法。首先,通过设置初始状态集合、决策选项、奖励函数、Q函数迭代方法等,建立了火箭任务决策分层强化学习模型,得到对火箭后续飞行进行评价的“评价网络”;然后利用基于凸优化的在线能力评估和轨迹规划方法,得到近似动态规划原理中的“决策生成”模块;最后,通过两者结合完成对火箭故障下后续飞行中连续轨迹和各级飞行段离散轨道根数等的决策。仿真结果表明该方法能够在非致命推力下降故障下实现火箭全程飞行任务决策并给出飞行轨迹。

基于互信息优化的Option-Critic算法

时序抽象作为分层强化学习的重要研究内容,允许分层强化学习智能体在不同的时间尺度上学习策略,可以有效解决深度强化学习难以处理的稀疏奖励问题。如何端到端地学习到优秀的时序抽象策略一直是分层强化学习研究面临的挑战。Option-Critic(OC)框架在Option框架的基础上,通过策略梯度理论,可以有效解决此问题。然而,在策略学习过程中,OC框架会出现Option内部策略动作分布变得十分相似的退化问题。该退化问题影响了OC框架的实验性能,导致Option的可解释性变差。为了解决上述问题,引入互信息知识作为内部奖励,并提出基于互信息优化的Option-Critic算法(Option-Critic Algorithm with Mutual Information Optimization,MIOOC)。MIOOC算法结合了近端策略Option-Critic(Proximal Policy Option-Critic,PPOC)算法,可以保证下层策略的多样性。为了验证算法的有效性,把MIOOC算法和几种常见的强化学习方法在连续实验环境中进行对比实验。实验结果表明,MIOOC算法可以加快模型学习速度,实验性能更优,Option内部策略更有区分度。

一种分层强化学习的知识推理方法

针对知识推理过程中,随着推理路径长度的增加,节点的动作空间急剧增长,使得推理难度不断提升的问题,提出一种分层强化学习的知识推理方法(knowledge reasoning method of hierarchical reinforcement learning,MutiAg-HRL),降低推理过程中的动作空间大小。MutiAg-HRL调用高级智能体对知识图谱中的关系进行粗略推理,通过计算下一步关系及给定查询关系之间的相似度,确定目标实体大致位置,依据高级智能体给出的关系,指导低级智能体进行细致推理,选择下一步动作;模型还构造交互奖励机制,对两个智能体的关系和动作选择及时给予奖励,防止模型出现奖励稀疏问题。为验证该方法的有效性,在FB15K-237和NELL-995数据集上进行实验,将实验结果与TransE、MINERVA、HRL等11种主流方法进行对比分析,MutiAg-HRL方法在链接预测任务上的hits@k平均提升了1.85%,MRR平均提升了2%。

基于兵棋推演的空战编组对抗智能决策方法

基于兵棋研究的空战编组对抗方法主要使用规则或运筹等手段,存在假设不够合理、建模不准确、应变性差等缺陷。强化学习算法可以根据作战数据自主学习编组对抗策略,以应对复杂的战场情况,但现有强化学习对作战数据要求高,当动作空间过大时,算法收敛慢,且对仿真平台有较高的要求。针对上述问题,提出了一种融合知识数据和强化学习的空战编组对抗智能决策方法,该决策方法的输入是战场融合态势,使用分层决策框架控制算子选择并执行任务,上层包含使用专家知识驱动的动作选择器,下层包含使用专家知识和作战规则细化的避弹动作执行器、侦察动作执行器和使用强化学习算法控制的打击动作执行器。最后基于典型作战场景进行实验,验证了该方法的可行性和实用性,且具有建模准确、训练高效的优点。

基于分层强化学习的多智能体博弈策略生成方法

典型基于深度强化学习的多智能体对抗策略生成方法采用“分总”框架,各智能体基于部分可观测信息生成策略并进行决策,缺乏从整体角度生成对抗策略的能力,大大限制了决策能力。为了解决该问题,基于分层强化学习提出改进的多智能体博弈策略生成方法。基于分层强化学习构建观测信息到整体价值的决策映射,以最大化整体价值作为目标构建优化问题,并推导了策略优化过程,为后续框架结构和方法实现的设计提供了理论依据;基于决策映射与优化问题构建,采用神经网络设计了模型框架,详细阐述了顶层策略控制模型和个体策略执行模型;基于策略优化方法,给出详细训练流程和算法流程;采用星际争霸多智能体对抗(StarCraft Multi-Agent Challenge,SMAC)环境,与典型多智能体方法进行性能对比。实验结果表明,该方法能够有效生成对抗策略,控制异构多智能体战胜预设对手策略,相比典型多智能体强化学习方法性能提升明显。

考虑威胁区通行概率的变体飞行器轨迹规划

针对威胁区交叉重叠且全覆盖飞行路径的变体飞行器轨迹规划问题,提出了一种考虑威胁区通行概率和变外形参数优化的轨迹规划方法。基于分层强化学习思想,通过设置飞行环境集合、决策选项、代价函数、Q函数以及选项内的策略等,建立了变体飞行器路径决策的分层强化学习模型;通过训练得到的评价网络,能够结合威胁区通行概率对实际的场景进行路径决策;根据飞行器可变外形的特点,优化所得决策结果的参数,得到全过程的通行轨迹和外形形态。仿真结果表明,该方法能够根据实际情况实时决策飞行路径,经过优化后得到全过程的优化轨迹和飞行形态。

基于分层强化学习的机器人自主避障算法仿真

智能机器人可以实时感知周围环境信息,通过绘制环境地图控制行动轨迹,但是如何自主导航起点到终点的同时避开障碍物,获取最优路径的问题仍然需要进一步解决。为提高机器人路径规划能力,减少机器人与障碍物的碰撞概率,提出基于分层强化学习算法的机器人自主避障方法。结合机器人的移动速度、角速度等相关信息,建立运动学模型,分别确立局部和全局坐标系。通过坐标转换,采集机器人和障碍物信息,构建分层强化学习整体架构,分为环境信息交互、子任务选择和根任务协作三个层次。将Q学习方法作为强化学习策略,设定Q函数值更新规则。通过笛卡尔乘积形式表示环境状态信息,选取合理的奖赏函数,提高学习效率,通过赋予Q值最大化的方式控制机器人最佳动作,实现自主避障。实验测试结果验证了上述方法能够精准躲避静态和动态障碍物,计算复杂度较低,可避免陷入局部最优。

基于学习博弈和契约论的分层联邦学习隐私保护激励机制

分层联邦学习(hierarchical federated learning,HFL)旨在通过多层架构的协作学习,同时保护隐私和优化模型性能.但其效果需依赖于针对参与各方的有效激励机制及应对信息不对称的策略.为了解决上述问题,本文提出一种保护终端设备、边缘服务器及云服务器隐私的分层激励机制.在边端层,边缘服务器作为中介应用多维合约理论设计不同类型的契约项,促使终端设备在不泄露数据采集、模型训练以及模型传输成本的情况下,使用本地数据参与HFL.在云边层,云服务器与边缘服务器间关于单位数据奖励和数据量的关系通过Stackelberg博弈进行建模,在不泄露边缘服务器单位利润的情况下,进一步将其转化为马尔可夫过程,并采用保护隐私的多智能体深度强化学习(multi-agent deep reinforcement learning,MADRL)方法逐渐接近斯塔克伯格均衡(Stackelberg equilibrium,SE).实验结果表明,本文提出的分层激励机制在性能上优于基线方法,云服务器的收益提升了接近11%,单位成本获取增益提升接近18倍.

Adaptive state-constrained/model-free iterative sliding mode control for aerial robot trajectory tracking

This paper develops a novel hierarchical control strategy for improving the trajectory tracking capability of aerial robots under parameter uncertainties.The hierarchical control strategy is composed of an adaptive sliding mode controller and a model-free iterative sliding mode controller(MFISMC).A position controller is designed based on adaptive sliding mode control(SMC)to safely drive the aerial robot and ensure fast state convergence under external disturbances.Additionally,the MFISMC acts as an attitude controller to estimate the unmodeled dynamics without detailed knowledge of aerial robots.Then,the adaption laws are derived with the Lyapunov theory to guarantee the asymptotic tracking of the system state.Finally,to demonstrate the performance and robustness of the proposed control strategy,numerical simulations are carried out,which are also compared with other conventional strategies,such as proportional-integralderivative(PID),backstepping(BS),and SMC.The simulation results indicate that the proposed hierarchical control strategy can fulfill zero steady-state error and achieve faster convergence compared with conventional strategies.

基于强化学习的离场飞行程序航迹生成方法

现代飞行程序设计受地形、障碍物、空域和飞行性能等多种因素的影响,设计过程中需进行大量针对设计细节有效性的评估工作;设计完毕的飞行程序还需专业的飞行试飞人员进行模拟机和真机试飞,耗费大量的人力、经济成本。如果试飞前缺少针对性的分析评估,一方面会增加试飞成本的支出,另一方面也会导致真机试飞环节存在安全隐患。针对上述问题,利用深度强化学习技术,提出一种在满足飞行程序设计规范条件下,面向飞行程序有效性和可行性验证的离场航迹自动生成方法。首先,利用空气动力学原理,建立考虑飞行性能和障碍物超障因素的基本飞行动力学模型,并借助Unity3D引擎构建三维可视化的训练平台;其次,在PyTorch深度学习框架中,利用Mlagents强化学习平台构建航空器在飞行时各个阶段的试飞训练模型,设计包括起飞、转弯、巡航和降落这4个目标的场景和奖励函数。以离场飞行程序试飞为例,采用厦门高崎机场某PBN(Performance Based Navigation)离场程序进行实例训练验证,并利用动态时间规整(DTW)距离量化实际生成航迹与标称航迹之间的偏离度。实验结果显示,偏差度满足飞行程序超障保护区的限制要求。上述训练模型在其他离场程序的实验结果也验证了模型具有较好的泛化能力。

基于脉冲神经网络的指挥智能体技术

针对现有智能体技术应用于军事指挥控制领域中时存在计算资源需求高、奖励值稀疏、收敛速度慢、推理效果差的问题,提出了一种基于脉冲神经网络(spiking neural network,SNN)和分层强化学习的指挥智能体技术。基于分层强化学习思想对军事指挥智能体进行建模,利用SNN构建智能体决策模型;通过ANN-SNN转换的学习算法获得基于SNN的指挥智能体;基于“墨子”兵棋推演软件开展对比试验,与现有智能体技术相比,提出方法对计算资源的需求较低,且具有较高的博弈对抗胜率。