Deep reinforcement learning using least-squares truncated temporal-difference

Policy evaluation(PE)is a critical sub-problem in reinforcement learning,which estimates the value function for a given policy and can be used for policy improvement.However,there still exist some limitations in current PE methods,such as low sample efficiency and local convergence,especially on complex tasks.In this study,a novel PE algorithm called Least-Squares Truncated Temporal-Difference learning(LST2D)is proposed.In LST2D,an adaptive truncation mechanism is designed,which effectively takes advantage of the fast convergence property of Least-Squares Temporal Difference learning and the asymptotic convergence property of Temporal Difference learning(TD).Then,two feature pre-training methods are utilised to improve the approximation ability of LST2D.Furthermore,an Actor-Critic algorithm based on LST2D and pre-trained feature representations(ACLPF)is proposed,where LST2D is integrated into the critic network to improve learning-prediction efficiency.Comprehensive simulation studies were conducted on four robotic tasks,and the corresponding results illustrate the effectiveness of LST2D.The proposed ACLPF algorithm outperformed DQN,ACER and PPO in terms of sample efficiency and stability,which demonstrated that LST2D can be applied to online learning control problems by incorporating it into the actor-critic architecture.

Efficient Temporal Difference Learning with Adaptive λ

Aim To find a more efficient learning method based on temporal difference learning for delayed reinforcement learning tasks. Methods A kind of Q learning algorithm based on truncated TD( λ ) with adaptive schemes of λ value selection addressed to absorbing Markov decision processes was presented and implemented on computers. Results and Conclusion Simulations on the shortest path searching problems show that using adaptive λ in the Q learning based on TTD( λ ) can speed up its convergence.

A Collaborative Machine Learning Scheme for Traffic Allocation and Load Balancing for URLLC Service in 5G and Beyond

Key challenges for 5G and Beyond networks relate with the requirements for exceptionally low latency, high reliability, and extremely high data rates. The Ultra-Reliable Low Latency Communication (URLLC) use case is the trickiest to support and current research is focused on physical or MAC layer solutions, while proposals focused on the network layer using Machine Learning (ML) and Artificial Intelligence (AI) algorithms running on base stations and User Equipment (UE) or Internet of Things (IoT) devices are in early stages. In this paper, we describe the operation rationale of the most recent relevant ML algorithms and techniques, and we propose and validate ML algorithms running on both cells (base stations/gNBs) and UEs or IoT devices to handle URLLC service control. One ML algorithm runs on base stations to evaluate latency demands and offload traffic in case of need, while another lightweight algorithm runs on UEs and IoT devices to rank cells with the best URLLC service in real-time to indicate the best one cell for a UE or IoT device to camp. We show that the interplay of these algorithms leads to good service control and eventually optimal load allocation, under slow load mobility. .

基于生成模型的Q-learning二分类算法

对于二分类问题,基于判别模型的分类器一般都是寻找一条最优判决边界,容易受到数据波动的影响。针对该问题提出一种基于生成模型的Q-learning二分类算法(BGQ-learning),将状态和动作分开编码,得到对应各类的判决函数,增加了决策空间的灵活性,同时在求解参数时,采用最小二乘时序差分(TD)算法和半梯度下降法的组合优化方法,加速了参数的收敛速度。设计实验对比了BGQ-learning算法与三种经典分类器以及一种新颖的分类器的分类性能,在UCI数据库七个数据集上的测试结果表明,该算法有着优良的稳定性以及良好的分类精确度。

Incremental Multi Step R Learning

Aim To investigate the model free multi step average reward reinforcement learning algorithm. Methods By combining the R learning algorithms with the temporal difference learning (TD( λ ) learning) algorithms for average reward problems, a novel incremental algorithm, called R( λ ) learning, was proposed. Results and Conclusion The proposed algorithm is a natural extension of the Q( λ) learning, the multi step discounted reward reinforcement learning algorithm, to the average reward cases. Simulation results show that the R( λ ) learning with intermediate λ values makes significant performance improvement over the simple R learning.

Balance Control of a Biped Robot on a Rotating Platform Based on Efficient Reinforcement Learning

In this work,we combined the model based reinforcement learning(MBRL)and model free reinforcement learning(MFRL)to stabilize a biped robot(NAO robot)on a rotating platform,where the angular velocity of the platform is unknown for the proposed learning algorithm and treated as the external disturbance.Nonparametric Gaussian processes normally require a large number of training data points to deal with the discontinuity of the estimated model.Although some improved method such as probabilistic inference for learning control(PILCO)does not require an explicit global model as the actions are obtained by directly searching the policy space,the overfitting and lack of model complexity may still result in a large deviation between the prediction and the real system.Besides,none of these approaches consider the data error and measurement noise during the training process and test process,respectively.We propose a hierarchical Gaussian processes(GP)models,containing two layers of independent GPs,where the physically continuous probability transition model of the robot is obtained.Due to the physically continuous estimation,the algorithm overcomes the overfitting problem with a guaranteed model complexity,and the number of training data is also reduced.The policy for any given initial state is generated automatically by minimizing the expected cost according to the predefined cost function and the obtained probability distribution of the state.Furthermore,a novel Q(λ)based MFRL method scheme is employed to improve the policy.Simulation results show that the proposed RL algorithm is able to balance NAO robot on a rotating platform,and it is capable of adapting to the platform with varying angular velocity.

An Adaptive Strategy via Reinforcement Learning for the Prisoner＇s Dilemma Game

The iterated prisoner's dilemma(IPD) is an ideal model for analyzing interactions between agents in complex networks. It has attracted wide interest in the development of novel strategies since the success of tit-for-tat in Axelrod's tournament. This paper studies a new adaptive strategy of IPD in different complex networks, where agents can learn and adapt their strategies through reinforcement learning method. A temporal difference learning method is applied for designing the adaptive strategy to optimize the decision making process of the agents. Previous studies indicated that mutual cooperation is hard to emerge in the IPD. Therefore, three examples which based on square lattice network and scale-free network are provided to show two features of the adaptive strategy. First, the mutual cooperation can be achieved by the group with adaptive agents under scale-free network, and once evolution has converged mutual cooperation, it is unlikely to shift. Secondly, the adaptive strategy can earn a better payoff compared with other strategies in the square network. The analytical properties are discussed for verifying evolutionary stability of the adaptive strategy.

TD再励学习在卫星姿态控制中的应用

随着卫星姿态控制系统对控制精度、鲁棒性和抗干扰要求的不断提高,将模糊神经网络控制引入到三轴稳定卫星的姿态控制中,并采用基于时差(TD)法的再励学习来解决模糊神经网络参数在线调整的问题,可以在无需训练样本的前提下实现控制器的在线学习.仿真结果表明,这种结合再励学习的控制算法不仅可以满足对姿态控制精度的要求,有效地抵制了外界干扰,并对卫星的不确定性有较强的鲁棒性.

基于优先采样模型的离线强化学习

离线强化学习通过减小分布偏移实现了习得策略向行为策略的逼近,但离线经验缓存的数据分布往往会直接影响习得策略的质量.通过优化采样模型来改善强化学习智能体的训练效果,提出两种离线优先采样模型:基于时序差分误差的采样模型和基于鞅的采样模型.基于时序差分误差的采样模型可以使智能体更多地学习值估计不准确的经验数据,通过估计更准确的值函数来应对可能出现的分布外状态.基于鞅的采样模型可以使智能体更多地学习对策略优化有利的正样本,减少负样本对值函数迭代的影响.进一步,将所提离线优先采样模型分别与批约束深度Q学习(Batch-constrained deep Q-learning,BCQ)相结合,提出基于时序差分误差的优先BCQ和基于鞅的优先BCQ.D4RL和Torcs数据集上的实验结果表明:所提离线优先采样模型可以有针对性地选择有利于值函数估计或策略优化的经验数据,获得更高的回报.

梯级水库深度强化学习长期随机优化调度研究

梯级水库调度相较于单库调度状态空间呈指数级增大,为解决基于表格的强化学习方法在解决梯级水库长期随机优化调度问题时面临的维数灾问题,提出采用深度强化学习中的深度Q网络算法求解。首先基于Copula函数分析梯级水库随机入库径流的联合分布函数;再根据时序差分思想分别建立目标神经网络和主神经网络,分别逼近当前和下一状态对应的动作状态价值,并采用ε-贪婪探索利用策略获取最优调度策略;最后将主要参数分步调优保障调度效益。算例对比表明,深度Q网络算法相较于Q学习算法及其改进算法提升了优化调度目标值,加快收敛速度,有效解决了梯级水库随机优化调度中的维数灾问题.

基于深度强化学习的动态装配算法

针对动态装配环境中存在的复杂、动态的噪声扰动,提出一种基于深度强化学习的动态装配算法。将一段时间内的接触力作为状态,通过长短时记忆网络进行运动特征提取;定义序列贴现因子,对之前时刻的分奖励进行加权得到当前时刻的奖励值;模型输出的动作为笛卡尔空间位移,使用逆运动学调整机器人到达期望位置。与此同时,提出一种对带有资格迹的时序差分算法改进的神经网络参数更新方法,可缩短模型训练时间。在实验部分,首先在圆孔–轴的简单环境中进行预训练,随后在真实场景下继续训练。实验证明提出的方法可以很好地适应动态装配任务中柔性、动态的装配环境。

基于时序差分的动态信道分配算法

合理的信道分配可以提高移动蜂窝网络的服务质量,用户高速移动以及流量突增等应用场景给移动蜂窝网络信道分配技术带来了挑战,动态信道分配算法成为当前研究的热点。基于此,首先引入用户呼叫的移动模型及其相关定义;其次建立了移动蜂窝网络信道分配环境的马尔可夫模型,使用时序差分对问题模型求解;最后通过搭建移动蜂窝网络信道分配仿真平台,在小区均匀流量和不均匀流量分布场景下对不同信道分配算法进行仿真实验。结果表明,所提算法大大降低了切换呼叫阻塞率,满足高移动高流量通信场景下的用户需求。

基于深度强化学习的非置换流水车间调度问题

针对传统调度算法不能有效利用历史数据进行学习,实时性较差而难以应对复杂多变的实际生产调度环境等问题,首次提出一种基于时序差分法的深度强化学习算法。该方法综合神经网络和强化学习实时性、灵活性的优势,直接依据输入的加工状态进行行为策略选取,更贴近实际订单响应式生产制造系统的调度决策过程。通过将调度问题转化为多阶段决策问题,用深度神经网络模型拟合状态值函数,把制造系统加工状态特征数据输入模型,采用时序差分法训练模型,把启发式算法或分配规则作为调度决策候选行为,结合强化学习在线评价—执行机制,从而为每次调度决策选取最优组合行为策略。在非置换流水车间标准问题集上的测试结果表明,该算法能够取得低于实例上界的较优解。

增强型深度确定策略梯度算法

针对深度确定策略梯度算法收敛速率较慢的问题,提出了一种增强型深度确定策略梯度(E-DDPG)算法。该算法在深度确定策略梯度算法的基础上,重新构建两个新的样本池——多样性样本池和高误差样本池。在算法执行过程中,训练样本分别从多样性样本池和高误差样本池按比例选取,以兼顾样本多样性以及样本价值信息,提高样本的利用效率和算法的收敛性能。此外,进一步从理论上证明了利用自模拟度量方法对样本进行相似性度量的合理性,建立值函数与样本相似性之间的关系。将E-DDPG算法以及DDPG算法用于经典的Pendulum问题和MountainCar问题,实验结果表明,E-DDPG具有更好的收敛稳定性,同时具有更快的收敛速率。

强化学习的模型、算法及应用

强化学习不需要具有先验知识,通过试错与环境交互获得策略的改进,具有自学习和在线学习能力,是构造智能体的核心技术之一。文中首先综述了强化学习模型和基本原理,然后介绍了强化学习的主要算法,包括Sarsa算法、TD算法、Q-学习算法及函数估计算法,最后介绍了强化学习的应用情况及未来研究方向。

面向机器博弈的即时差分学习研究

以六子棋机器博弈为应用背景,实现了基于即时差分学习的估值函数权值调整自动化。提出了一种新的估值函数设计方案,解决了先验知识与多层神经元网络结合的问题。结合具体应用对象的特性,提出了对即时差分序列进行选择性学习的方法,在一定程度上避免了无用状态的干扰。经过10020盘的自学习训练,与同一个程序对弈,其胜率提高了8%左右,具有良好的效果。

基于递推最小二乘法的多步时序差分学习算法

强化学习是一种重要的机器学习方法。为了提高强化学习过程的收敛速度和减少学习过程值函数估计的误差,提出了基于递推最小二乘法的多步时序差分学习算法(RLS-TD(λ))。证明了在满足一定条件下,该算法的权值将以概率1收敛到唯一解,并且得出和证明了值函数估计值的误差应满足的关系式。迷宫实验表明,与RLS-TD(0)算法相比,该算法能加快学习过程的收敛,与传统的TD(λ)算法相比,该算法减少了值函数估计误差,从而提高了精度。

基于二阶时序差分误差的双网络DQN算法

针对深度Q网络(DQN)算法因过估计导致收敛稳定性差的问题,在传统时序差分(TD)的基础上提出N阶TD误差的概念,设计基于二阶TD误差的双网络DQN算法。构造基于二阶TD误差的值函数更新公式,同时结合DQN算法建立双网络模型,得到两个同构的值函数网络分别用于表示先后两轮的值函数,协同更新网络参数,以提高DQN算法中值函数估计的稳定性。基于Open AI Gym平台的实验结果表明,在解决Mountain Car和Cart Pole问题方面,该算法较经典DQN算法具有更好的收敛稳定性。

加强学习主要算法的比较研究

文章介绍了加强学习模型,分别给出了加强学习的四个主要算法:动态规划、蒙特卡罗算法、时序差分算法、Q-学习,并指出了它们之间的区别和联系。最后给出加强学习的两个应用以及今后的研究方向。

基于时间差分和局部加权偏最小二乘算法的过程自适应软测量建模

工业过程软测量模型常常因为过程的变量漂移、非线性和时变等问题而使得预测性能下降。因此,时间差分已被应用于解决过程变量漂移问题。但是,时间差分框架下的全局模型往往不能很好地描述过程非线性和时变等特性。为此,提出了一种融合时间差分模型和局部加权偏最小二乘算法的自适应软测量建模方法。时间差分模型可以大大减少过程变量漂移的影响,而局部加权偏最小二乘算法作为一种即时学习方法,可以有效解决过程非线性和时变问题。该方法的有效性在数值例子和工业过程实例中得到了有效验证。