基于线性动态跳帧的深度双Q网络

Deep Double Q-Network Based on Linear Dynamic Frame Skip

深度Q网络模型在处理需要感知高维输入数据的决策控制任务中性能良好.然而,在深度Q网络及其改进算法中基本使用静态的跳帧方法,即动作被重复执行固定的次数.另外,优先级经验重放是对均匀采样的一种改进,然而目前各个研究仅将样本的时间差分误差作为评价优先级的标准.针对这两个问题,该文提出一种基于线性动态跳帧和改进的优先级经验重放的深度双Q网络.该算法使得跳帧率成为一个可动态学习的参数,跳帧率随网络输出Q值的大小线性增长,Agent将根据当前状态和动作来动态地确定一个动作被重复执行的次数,并利用经验池中样本的每个动作的跳帧率和样本的时间差分误差共同决定样本的优先级.最后在Atari 2600游戏中进行实验,结果表明该算法相比于传统动态跳帧和优先级经验重放算法具有更优的效果.

Deep Q-Network is able to perform human-level control for handling problems requiring both rich perception of high-dimensional raw inputs and policy control.However,the current state of the art architectures like Deep Q-Network and it improved algorithms adopt a traditional framework with a static frame skip rate,where the action output from the network is repeated for a fixed number of frames regardless of the current state.Although Dynamic Frame skip Deep Q-Network uses a dynamic frame skip rate,it doubles the number of nodes in the network output layer with a frame skip rate of 4 or 20.Such settings may cause an increase in the amount of computation of the network,and cause bad actions to be performed multiple times,thereby affecting the efficiency of learning.In addition,an important technique in Deep Q-Network is the use of an experience replay mechanism.The traditional method of uniformly sampling samples ignores the importance of samples.In order to increase the sampling rate of important samples,the prioritized experience replay is an improvement on uniform sampling,using only the temporal difference error of a sample as the criterion of evaluation priority.However,such priority evaluation criterion only considers the temporal difference error of the sample,and there may be other factors that affect the priority of the sample.In this paper,we propose a new algorithm:Deep Double Q-Network based on Linear Dynamic Frame Skip and Improved Prioritized Experiential Replay(LDF-IPER-DDQN in short).The value of the frame-skip rate increases linearly with the magnitude of the network output Q value,which allows Agent to dynamically select the number of times an action is repeated based on the current state and action.For the action with the largest Q value,the maximum frame skip rate is given to this action.In contrast,the action with the smallest Q value is given the minimum frame skip rate.In this way,the frame skip rate becomes a dynamic learnable parameter.Furthermore,the value of the frame skip rate for each action in the experience pool is considered as another factor to evaluate the sampling priority.The priority of the sample is determined by the frame skip rate of each action of the sample in the experience pool and the temporal difference error of the sample.As a sequence,if two transition samples have similar temporal difference errors,the transition samples with larger frame skip rates are replayed more frequently.In the experiment of this paper,we evaluate the performance of the new algorithm through eight challenging strategic games with sparse reward from the set of classic Atari 2600 games.They are Seaquest,Assault,Asterix,Q*bert,SpaceInvaders,Berzerk,BeamRider and Gopher.We evaluate the performance of the new algorithm during the training and testing phases.The experimental results show that LDF-IPER-DDQN performs better than some traditional deep reinforcement learning algorithms on these Atari 2600 games.It achieves better performance in terms of the average reward per episode and has better stability on these games.