兼顾通信效率与效用的自适应高斯差分隐私个性化联邦学习

Communication-efficient and Utility-A ware Adaptive Gaussian Differential Privacy for Personalized Federated Learning

近年来,由于联邦学习中的通信参数(或梯度)会给参与方本地敏感数据带来重大的隐私泄露风险,联邦学习隐私保护引起了广泛的关注.然而,梯度交换频繁、数据分布异构、参与方本地硬件资源受限等一系列不可避免的因素给联邦学习隐私保护增加了挑战难度.为了以一种统一的方式同时有效地解决数据隐私、模型效用、通信效率以及参与方数据非独立同分布等四个方面的问题,本文提出了一种新的兼顾通信效率与效用的自适应高斯差分隐私个性化联邦学习(Communication-efficient and Utility-aware Adaptive Gaussian Differential Privacy for Personalized Federated Learning,CUAG-PFL)方法.具体而言,本文提出一种动态层级压缩模型梯度的方案先为通信模型梯度每一层动态生成特定的压缩率,再根据压缩率构造对应的确定性二进制测量矩阵去除梯度冗余信息.随后,通过同时优化裁剪阈值、敏感度和噪声尺度等隐私相关参数来对压缩的模型梯度执行自适应高斯差分隐私操作.此外,本文对CUAG-PFL进行了严格的隐私分析.为了验证CUAG-PFL在隐私、效用、通信效率以及个性化四个方面的优势,本文在CIFAR-10和CIFAR-100两个真实联邦数据集上进行了大量实验模拟、对比和分析,结果表明CUAG-PFL能够提高参与方本地数据隐私性、通信效率和模型效用,同时解决了数据非独立同分布的问题.特别地,即使在隐私预算仅为0.92且上行通信量减少68.6%时,CUAG-PFL因隐私保护和梯度压缩所引起的模型效用损失仅为1.66%.

In recent years,there has been an increasing focus on the privacy protection in the field of federated learning(FL).This widespread attention is mainly due to the fact that commu-nication parameters(or gradients)during the process of collaborative learning among the central server and various participants can cause the significant risk of the privacy leakage.In other words,the communication process in the FL system poses a potential threat of exposing the sensitive data belonging to local participants,which has raised heightened concerns among researchers and practitioners.Furthermore,in addition to the challenge of the privacy protection in FL,a series of other unavoidable factors such as the frequent gradients exchange,the hetero-geneous data distribution among local participants,and limited resources available on the local hardware need to be simultaneously taken into consideration.These factors obviously add diffi-culties to the challenge of the privacy protection in FL.In order to effectively address four critical issues of data privacy,model utility,communication efficiency,and non-independently and iden-tically distributed data among local participants in a unified manner,this paper proposes a novel Communication-efficient and Utility-aware Adaptive Gaussian Differential Privacy for Personal-ized FL method,called CUAG-PFL.Specifically,a dynamic layer-compression scheme for model gradients in the FL system is proposed.This scheme aims to improve the communication efficien-cy as much as possible and reduce the loss of the model utility caused by compression and recon-struction through dynamically customizing the compression rate for each layer of communication gradients,and then constructing the corresponding deterministic binary measurement matrix based on the compression rate.This designed deterministic binary measurement matrix can effec-tively remove the redundant information of model gradients that needs to be uploaded to the cen-tral server.Subsequently,the adaptive Gaussian differential privacy operation is performed on compressed model gradients of local participants.This operation involves optimizing the main privacy-related parameters such as the clipping threshold,the sensitivity,and the noise scale.By optimizing these parameters at the same time,this operation ensures that the privacy of the local data is preserved,while allowing each model of the corresponding local participant to have the satisfactory performance.In addition,the rigorous privacy analysis of the proposed CUAG-PFL is presented in this paper.In order to validate the superiority of the proposed CUAG-PFL in four critical aspects of data privacy,model utility,communication efficiency,and personalization,a large number of experimental simulations,comparisons,and analyses are conducted on two classic real-world federated datasets,i.e.,CIFAR-10 and CIFAR-100.All experimental results and analyses show that the proposed CUAG-PFL can simultaneously improve the privacy of local sensitive data,the communication efficiency and the model utility,as well as address the problem of non-independently and identically distributed data among local participants in the FL system.In particular,it is worth emphasizing that even when the privacy budget is only 0.92 and the amount of the upstream communication is reduced by 68.6%,the loss of the model performance caused by both the privacy protection and the communication gradients compression is just 1.66%for the proposed CUAG-PFL.