基于梯度的深度强化学习解释方法

Gradient-based Deep Reinforcement Learning Interpretation Methods

DQN等深度强化学习方法的学习过程与工作机制不透明,无法感知其决策依据与决策可靠性,使模型做出的决策饱受质疑,极大限制了深度强化学习的应用场景。为了解释智能体的决策机理,提出一种基于梯度的显著性图生成算法(saliency map generation algorithm based on gradient,SMGG)。使用高层卷积层生成的特征图梯度信息计算不同特征图的重要性,在模型的结构和内部参数已知的情况下,从模型最后一层入手,通过对特征图梯度的计算,生成不同特征图相对于显著性图的权重;对特征重要性进行正向和负向分类,利用有正向影响的权值将特征图中捕获的特征进行加权,构成当前决策的正向解释;利用对其他类别有负向影响的权值将特征图中捕获的特征进行加权,构成当前决策的反向解释。二者共同生成决策的显著性图,得出智能体决策行为的依据,实验证明了该方法的有效性。

The learning process and working mechanism of deep reinforcement learning methods such as DQN are not transparent,and their decision basis and reliability cannot be perceived,which makes the decisions made by the model highly questionable and greatly limits the application scenarios of deep reinforcement learning.To explain the decision-making mechanism of intelligent agents,this paper proposes a gradient based saliency map generation algorithm SMGG.It uses the gradient information of feature maps generated by high-level convolutional layers to calculate the importance of different feature maps.With the known structure and internal parameters of the model,starting from the last layer of the model,the weight of different feature maps relative to the saliency map is generated by calculating the gradient of feature maps;it classifies the importance of features in both positive and negative directions,and uses weights with positive influence to weight the features captured in the feature map,forming a positive interpretation of the current decision;it uses weights that have a negative impact on other categories to weight the features captured in the feature map,forming a reverse interpretation of the current decision.The saliency map of the decision is generated by the two together,and the basis for the intelligent agent's decision-making behavior is obtained.The effectiveness of this method has been demonstrated through experiments.