基于双重注意力机制的异步优势行动者评论家算法

Asynchronous Advantage Actor-Critic with Double Attention Mechanisms

深度强化学习是目前机器学习领域发展最快的技术之一.传统的深度强化学习方法在处理高维度大状态的空间任务时,庞大的计算量导致其训练时间过长.虽然异步深度强化学习利用异步方法极大缩短了训练时间,但会忽略某些更具价值的图像区域和图像特征.针对上述问题,本文提出了一种基于双重注意力机制的异步优势行动者评论家算法.新算法利用特征注意力机制和视觉注意力机制来改进传统的异步深度强化学习模型.其中,特征注意力机制为卷积神经网络卷积后的所有特征图设置不同的权重,使得智能体聚焦于重要的图像特征;同时,视觉注意力机制为图像不同区域设置权重参数,权重高的区域表示该区域信息对智能体后续的策略学习有重要价值,帮助智能体更高效地学习到最优策略.新算法引入双重注意力机制,从表层和深层两个角度对图像进行编码表征,帮助智能体将聚焦点集中在重要的图像区域和图像特征上.最后,通过Atari 2600部分经典实验验证了基于双重注意力机制的异步优势行动者评论家算法的有效性.

In recent years,deep reinforcement learning(DRL),which combines deep learning and reinforcement learning together,is a new research hotspot in artificial intelligence.As DRL takes advantage of deep learning,it is able to take raw images as input,which extends applications of reinforcement learning.At the mean while time,DRL retains the advantages of reinforcement learning in application such as intelligent policy decision or robotic control.However,traditional DRL such as deep Q-network(DQN)or double deep Q-network(DDQN),could hardly deal with complex tasks with high-dimensional state in a short time.Researchers have proposed many methods to solve this problem,and asynchronous advantage actor-critic(A3 C)is one of the most used algorithm.As we know,traditional asynchronous deep reinforcement learning can use multi-threading techniques to reduce large amounts of training time.However,when it comes to high-dimensional large-state space tasks,some valuable and important image areas and features are often ignored,such as Atari 2600 games.The reason is that Agent’s attention is focused on the entire input image and all features of the image,without any emphases on some important features.To handle this problem,we employ the attention mechanism to ameliorate the performance of traditional asynchronous deep reinforcement learning models.In recent years,inspired by human vision,the attention mechanism has been extensively used in machine translation,image recognition and speech recognition,becoming one of the most noteworthy and in-depth research techniques in the area of deep learning technologies.Based on this,we put forward an asynchronous advantage actor-critic with double attention mechanisms(DAM-A3 C).In DAM-A3 C,there are two main characteristics:visual attention mechanism(VAM)and feature attention mechanism(FAM).First,the application of visual attention mechanism can enable Agent to adaptively engage in the image region,especially in those more important areas which can enhance the cumulative reward at each moment,reducing the computational cost of the network’s training and finally accelerating the process of learning the approximate optimal strategy.Second,via the exertion of FAM,an asynchronous advantage actor-critic is expected to pay more attention to those features with more value.What we know is that different convolution kernels can generate different feature maps by operating convolution on the image in convolutional neural network.And feature maps completely describe the image from different features.The traditional training of convolutional neural network treats each extracted feature equally,which means all features have the same proportion,instead of different levels of focus according to their value.However,some image features have a crucial role in the description of images,such as color features,shape features and spatial relationship features,etc.In order to alleviate this problem,FAM can assist Agent to converge on feature maps with rich values,which will facilitate Agent to make correct decisions.To sum up,we introduce FAM in VAM-A3 C model and propose DAM-A3 C model.DAM-A3 Cutilizes visual attention mechanism and feature attention mechanism to enable Agent to concentrate on the important areas and important features of the image,which advances the network model to recognize important information and key features of the image in a short time.We select some classic Atari 2600 games as experimental objects to evaluate the performance of the new model.The experimental result shows that the new model has better performance than the traditional asynchronous advantage actor-critic algorithm in experimental tasks.