GenFedRL:面向深度强化学习智能体的通用联邦强化学习框架

GenFedRL:a general federated reinforcement learning framework for deep reinforcement learning agents

针对智能物联网中,搭载深度强化学习智能体的智能设备缺乏有效安全数据共享机制的问题,提出一种面向深度强化学习智能体的通用联邦强化学习(GenFedRL)框架。GenFedRL不需要共享深度强化学习智能体的本地私有数据,而通过模型共享技术实现共同训练,在保护各智能体私有数据隐私的同时,有效地利用其数据资源和计算资源。为应对现实通信环境的复杂性与满足加速训练的需要,为GenFedRL设计了基于同步并行的模型共享机制。结合常见深度强化学习算法自身的模型结构特点,基于FedAvg算法设计了适用于单网络结构与多网络结构的通用联邦强化学习算法,进而实现了具有同种网络结构的智能体间的模型共享机制,更好地保护各类智能体的私有数据。仿真实验表明,即使在大部分数据节点无法参与训练的恶劣通信环境下,常见深度强化学习算法智能体在所提框架上仍表现出良好的性能。

To solve the problem that intelligent devices equipped with deep reinforcement learning agents lack effective security data sharing mechanisms in the intelligent Internet of things,a general federated reinforcement learning(Gen-FedRL)framework was proposed for deep reinforcement learning agents.The joint training through model-sharing technology was realized by GenFedRL without the need to share the local private data of deep reinforcement learning agents.Each agent device’s data and computing resources could be effectively used without disclosing the privacy of its private data.To cope with the complexity of the real communication environment and meet the need to accelerate the training speed,a model-sharing mechanism based on synchronization and parallel was designed for GenFedRL.Combined with the model structure characteristics of common deep reinforcement learning algorithms,general federated reinforcement learning algorithm suitable for single network structure and multi-network structure was designed based on the FedAvg algorithm,respectively.Then,the model sharing mechanism among agents with the same network structure was implemented to protect the private data of various agents better.Simulation experiments show that common deep reinforcement learning algorithms still perform well in GenFedRL even in the harsh communication environment where most data nodes cannot participate in training.