面向深度强化学习的鲁棒性增强方法

Robustness Enhancement Method for Deep Reinforcement Learning

深度强化学习(Deep Reinforcement Learning, DRL)结合了深度学习的感知能力和强化学习的决策能力,被应用于许多领域.然而,一旦攻击者窃取了DRL数据,就能干扰状态、奖励及动作或环境,从而影响智能体的决策.且已有研究表明DRL模型极易受到恶意攻击,攻击者根据状态及动作空间信息,训练等价模型实现黑盒攻击.为了实现DRL数据隐私保护及模型鲁棒性增强,本文提出一种基于垂直联邦的DRL模型(Vertical Federated based DRL,VF-DRL).VF-DRL搭建多个客户端并保证数据特征不重叠.同时服务器端上传各个客户端输出的隐层特征以保证数据隐私.进一步,本文对比不同基线算法,通过大量实验评估了VF-DRL模型的性能.假设存在一个恶意客户端执行对抗攻击的情况下,使用多种对抗攻击方法验证了VF-DRL模型的鲁棒性.同时在高维及较低维环境中验证VF-DRL模型的鲁棒性,并进一步分析影响其鲁棒性的因素.

Deep Reinforcement Learning(DRL)combines the perception ability of deep learning with the decision-making ability of reinforcement learning and has been applied in many fields.However,if an attacker steals DRL data,interference with state,reward,and action or environment can be easily implemented to affect agent decisions.Moreover,previous studies have shown that DRL models are vulnerable to malicious attacks,so equivalent models can be trained to achieve black box attacks according to state and action space information.To realize DRL data protection and model robustness enhancement,this paper proposes a DRL model based on Vertical federation.Considering the privacy of data,a robust DRL model is established by combining vertical Federated based DRL,namely VF-DRL.VF-DRL builds multiple clients and ensures that data features do not overlap.Meanwhile,the server uploads hidden layer features output by each client to ensure data privacy.Further,this paper compares different baseline algorithms and evaluates the performance of the VF-DRL model through a large number of experiments.The robustness of the VF-DRL model is verified by using a variety of adversarial attack methods when a malicious client executes an adversarial attack.At the same time,the robustness of the VF-DRL model is verified in high-dimensional and low-dimensional environments,and the factors affecting its robustness are further analyzed.