Sharing data while protecting privacy in the industrial Internet is a significant challenge.Traditional machine learning methods require a combination of all data for training;however,this approach can be limited by d...Sharing data while protecting privacy in the industrial Internet is a significant challenge.Traditional machine learning methods require a combination of all data for training;however,this approach can be limited by data availability and privacy concerns.Federated learning(FL)has gained considerable attention because it allows for decentralized training on multiple local datasets.However,the training data collected by data providers are often non-independent and identically distributed(non-IID),resulting in poor FL performance.This paper proposes a privacy-preserving approach for sharing non-IID data in the industrial Internet using an FL approach based on blockchain technology.To overcome the problem of non-IID data leading to poor training accuracy,we propose dynamically updating the local model based on the divergence of the global and local models.This approach can significantly improve the accuracy of FL training when there is relatively large dispersion.In addition,we design a dynamic gradient clipping algorithm to alleviate the influence of noise on the model accuracy to reduce potential privacy leakage caused by sharing model parameters.Finally,we evaluate the performance of the proposed scheme using commonly used open-source image datasets.The simulation results demonstrate that our method can significantly enhance the accuracy while protecting privacy and maintaining efficiency,thereby providing a new solution to data-sharing and privacy-protection challenges in the industrial Internet.展开更多
Federated learning(FL) is a machine learning paradigm for data silos and privacy protection,which aims to organize multiple clients for training global machine learning models without exposing data to all parties.Howe...Federated learning(FL) is a machine learning paradigm for data silos and privacy protection,which aims to organize multiple clients for training global machine learning models without exposing data to all parties.However,when dealing with non-independently identically distributed(non-ⅡD) client data,FL cannot obtain more satisfactory results than centrally trained machine learning and even fails to match the accuracy of the local model obtained by client training alone.To analyze and address the above issues,we survey the state-of-theart methods in the literature related to FL on non-ⅡD data.On this basis,a motivation-based taxonomy,which classifies these methods into two categories,including heterogeneity reducing strategies and adaptability enhancing strategies,is proposed.Moreover,the core ideas and main challenges of these methods are analyzed.Finally,we envision several promising research directions that have not been thoroughly studied,in hope of promoting research in related fields to a certain extent.展开更多
Federated learning(FL),a cutting-edge distributed machine learning training paradigm,aims to generate a global model by collaborating on the training of client models without revealing local private data.The co-occurr...Federated learning(FL),a cutting-edge distributed machine learning training paradigm,aims to generate a global model by collaborating on the training of client models without revealing local private data.The co-occurrence of non-independent and identically distributed(non-IID)and long-tailed distribution in FL is one challenge that substantially degrades aggregate performance.In this paper,we present a corresponding solution called federated dual-decoupling via model and logit calibration(FedDDC)for non-IID and long-tailed distributions.The model is characterized by three aspects.First,we decouple the global model into the feature extractor and the classifier to fine-tune the components affected by the joint problem.For the biased feature extractor,we propose a client confidence re-weighting scheme to assist calibration,which assigns optimal weights to each client.For the biased classifier,we apply the classifier re-balancing method for fine-tuning.Then,we calibrate and integrate the client confidence re-weighted logits with the re-balanced logits to obtain the unbiased logits.Finally,we use decoupled knowledge distillation for the first time in the joint problem to enhance the accuracy of the global model by extracting the knowledge of the unbiased model.Numerous experiments demonstrate that on non-IID and long-tailed data in FL,our approach outperforms state-of-the-art methods.展开更多
Distributed stochastic gradient descent and its variants have been widely adopted in the training of machine learning models,which apply multiple workers in parallel.Among them,local-based algorithms,including Local S...Distributed stochastic gradient descent and its variants have been widely adopted in the training of machine learning models,which apply multiple workers in parallel.Among them,local-based algorithms,including Local SGD and FedAvg,have gained much attention due to their superior properties,such as low communication cost and privacypreserving.Nevertheless,when the data distribution on workers is non-identical,local-based algorithms would encounter a significant degradation in the convergence rate.In this paper,we propose Variance Reduced Local SGD(VRL-SGD)to deal with the heterogeneous data.Without extra communication cost,VRL-SGD can reduce the gradient variance among workers caused by the heterogeneous data,and thus it prevents local-based algorithms from slow convergence rate.Moreover,we present VRL-SGD-W with an effectivewarm-up mechanism for the scenarios,where the data among workers are quite diverse.Benefiting from eliminating the impact of such heterogeneous data,we theoretically prove that VRL-SGD achieves a linear iteration speedup with lower communication complexity even if workers access non-identical datasets.We conduct experiments on three machine learning tasks.The experimental results demonstrate that VRL-SGD performs impressively better than Local SGD for the heterogeneous data and VRL-SGD-W is much robust under high data variance among workers.展开更多
车联网在智慧城市建设中扮演着不可或缺的角色,汽车不仅仅是交通工具,更是大数据时代信息采集和传输的重要载体.随着车辆采集的数据量飞速增长和人们隐私保护意识的增强,如何在车联网环境中确保用户数据安全,防止数据泄露,成为亟待解决...车联网在智慧城市建设中扮演着不可或缺的角色,汽车不仅仅是交通工具,更是大数据时代信息采集和传输的重要载体.随着车辆采集的数据量飞速增长和人们隐私保护意识的增强,如何在车联网环境中确保用户数据安全,防止数据泄露,成为亟待解决的难题.联邦学习采用“数据不动模型动”的方式,为保护用户隐私和实现良好性能提供了可行方案.然而,受限于采集设备、地域环境、个人习惯的差异,多台车辆采集的数据通常表现为非独立同分布(non-independent and identically distributed,non-IID)数据,而传统的联邦学习算法在non-IID数据环境中,其模型收敛速度较慢.针对这一挑战,提出了一种面向non-IID数据的车联网多阶段联邦学习机制,称为FedWO.第1阶段采用联邦平均算法,使得全局模型快速达到一个基本的模型准确度;第2阶段采用联邦加权多方计算,依据各车辆的数据特性计算其在全局模型中的权重,聚合后得到性能更优的全局模型,同时采用传输控制策略,减少模型传输带来的通信开销;第3阶段为个性化计算阶段,车辆利用各自的数据进行个性化学习,微调本地模型获得与本地数据更匹配的模型.实验采用了驾驶行为数据集进行实验评估,结果表明相较于传统方法,在non-IID数据场景下,FedWO机制保护了数据隐私,同时提高了算法的准确度.展开更多
个性化联邦学习侧重于为各客户端提供个性化模型,旨在提高对异构数据的处理性能,然而现有的个性化联邦学习算法大多以增加客户端参数量为代价提高个性化模型的性能,使计算变得复杂.为了解决此问题,文中提出基于稀疏正则双层优化的个性...个性化联邦学习侧重于为各客户端提供个性化模型,旨在提高对异构数据的处理性能,然而现有的个性化联邦学习算法大多以增加客户端参数量为代价提高个性化模型的性能,使计算变得复杂.为了解决此问题,文中提出基于稀疏正则双层优化的个性化联邦学习算法(Personalized Federated Learning Based on Sparsity Regularized Bi-level Optimization,pFedSRB),在客户端的个性化更新中引入l 1范数稀疏正则化,提升个性化模型的稀疏度,避免不必要的客户端参数更新,降低模型复杂度.将个性化联邦学习建模为双层优化问题,内层优化采用交替方向乘子法,可提高学习速度.在4个联邦学习基准数据集上的实验表明,pFedSRB在异构数据上表现出色,在提高模型性能的同时有效降低训练用时和空间成本.展开更多
As the scale of federated learning expands,solving the Non-IID data problem of federated learning has become a key challenge of interest.Most existing solutions generally aim to solve the overall performance improveme...As the scale of federated learning expands,solving the Non-IID data problem of federated learning has become a key challenge of interest.Most existing solutions generally aim to solve the overall performance improvement of all clients;however,the overall performance improvement often sacrifices the performance of certain clients,such as clients with less data.Ignoring fairness may greatly reduce the willingness of some clients to participate in federated learning.In order to solve the above problem,the authors propose Ada-FFL,an adaptive fairness federated aggregation learning algorithm,which can dynamically adjust the fairness coefficient according to the update of the local models,ensuring the convergence performance of the global model and the fairness between federated learning clients.By integrating coarse-grained and fine-grained equity solutions,the authors evaluate the deviation of local models by considering both global equity and individual equity,then the weight ratio will be dynamically allocated for each client based on the evaluated deviation value,which can ensure that the update differences of local models are fully considered in each round of training.Finally,by combining a regularisation term to limit the local model update to be closer to the global model,the sensitivity of the model to input perturbations can be reduced,and the generalisation ability of the global model can be improved.Through numerous experiments on several federal data sets,the authors show that our method has more advantages in convergence effect and fairness than the existing baselines.展开更多
The rapid expansion of artificial intelligence(AI)applications has raised significant concerns about user privacy,prompting the development of privacy-preserving machine learning(ML)paradigms such as federated learnin...The rapid expansion of artificial intelligence(AI)applications has raised significant concerns about user privacy,prompting the development of privacy-preserving machine learning(ML)paradigms such as federated learning(FL).FL enables the distributed training of ML models,keeping data on local devices and thus addressing the privacy concerns of users.However,challenges arise from the heterogeneous nature of mobile client devices,partial engagement of training,and non-independent identically distributed(non-IID)data distribution,leading to performance degradation and optimization objective bias in FL training.With the development of 5G/6G networks and the integration of cloud computing edge computing resources,globally distributed cloud computing resources can be effectively utilized to optimize the FL process.Through the specific parameters of the server through the selection mechanism,it does not increase the monetary cost and reduces the network latency overhead,but also balances the objectives of communication optimization and low engagement mitigation that cannot be achieved simultaneously in a single-server framework of existing works.In this paper,we propose the FedAdaSS algorithm,an adaptive parameter server selection mechanism designed to optimize the training efficiency in each round of FL training by selecting the most appropriate server as the parameter server.Our approach leverages the flexibility of cloud resource computing power,and allows organizers to strategically select servers for data broadcasting and aggregation,thus improving training performance while maintaining cost efficiency.The FedAdaSS algorithm estimates the utility of client systems and servers and incorporates an adaptive random reshuffling strategy that selects the optimal server in each round of the training process.Theoretical analysis confirms the convergence of FedAdaSS under strong convexity and L-smooth assumptions,and comparative experiments within the FLSim framework demonstrate a reduction in training round-to-accuracy by 12%–20%compared to the Federated Averaging(FedAvg)with random reshuffling method under unique server.Furthermore,FedAdaSS effectively mitigates performance loss caused by low client engagement,reducing the loss indicator by 50%.展开更多
基金This work was supported by the National Key R&D Program of China under Grant 2023YFB2703802the Hunan Province Innovation and Entrepreneurship Training Program for College Students S202311528073.
文摘Sharing data while protecting privacy in the industrial Internet is a significant challenge.Traditional machine learning methods require a combination of all data for training;however,this approach can be limited by data availability and privacy concerns.Federated learning(FL)has gained considerable attention because it allows for decentralized training on multiple local datasets.However,the training data collected by data providers are often non-independent and identically distributed(non-IID),resulting in poor FL performance.This paper proposes a privacy-preserving approach for sharing non-IID data in the industrial Internet using an FL approach based on blockchain technology.To overcome the problem of non-IID data leading to poor training accuracy,we propose dynamically updating the local model based on the divergence of the global and local models.This approach can significantly improve the accuracy of FL training when there is relatively large dispersion.In addition,we design a dynamic gradient clipping algorithm to alleviate the influence of noise on the model accuracy to reduce potential privacy leakage caused by sharing model parameters.Finally,we evaluate the performance of the proposed scheme using commonly used open-source image datasets.The simulation results demonstrate that our method can significantly enhance the accuracy while protecting privacy and maintaining efficiency,thereby providing a new solution to data-sharing and privacy-protection challenges in the industrial Internet.
文摘Federated learning(FL) is a machine learning paradigm for data silos and privacy protection,which aims to organize multiple clients for training global machine learning models without exposing data to all parties.However,when dealing with non-independently identically distributed(non-ⅡD) client data,FL cannot obtain more satisfactory results than centrally trained machine learning and even fails to match the accuracy of the local model obtained by client training alone.To analyze and address the above issues,we survey the state-of-theart methods in the literature related to FL on non-ⅡD data.On this basis,a motivation-based taxonomy,which classifies these methods into two categories,including heterogeneity reducing strategies and adaptability enhancing strategies,is proposed.Moreover,the core ideas and main challenges of these methods are analyzed.Finally,we envision several promising research directions that have not been thoroughly studied,in hope of promoting research in related fields to a certain extent.
基金supported by the National Natural Science Foundation of China(No.61702321)。
文摘Federated learning(FL),a cutting-edge distributed machine learning training paradigm,aims to generate a global model by collaborating on the training of client models without revealing local private data.The co-occurrence of non-independent and identically distributed(non-IID)and long-tailed distribution in FL is one challenge that substantially degrades aggregate performance.In this paper,we present a corresponding solution called federated dual-decoupling via model and logit calibration(FedDDC)for non-IID and long-tailed distributions.The model is characterized by three aspects.First,we decouple the global model into the feature extractor and the classifier to fine-tune the components affected by the joint problem.For the biased feature extractor,we propose a client confidence re-weighting scheme to assist calibration,which assigns optimal weights to each client.For the biased classifier,we apply the classifier re-balancing method for fine-tuning.Then,we calibrate and integrate the client confidence re-weighted logits with the re-balanced logits to obtain the unbiased logits.Finally,we use decoupled knowledge distillation for the first time in the joint problem to enhance the accuracy of the global model by extracting the knowledge of the unbiased model.Numerous experiments demonstrate that on non-IID and long-tailed data in FL,our approach outperforms state-of-the-art methods.
基金This research was partially supported by grants from the National Key Research and Development Program of China(No.2018YFC0832101)the National Natural Science Foundation of China(Grant Nos.U20A20229 and 61922073).
文摘Distributed stochastic gradient descent and its variants have been widely adopted in the training of machine learning models,which apply multiple workers in parallel.Among them,local-based algorithms,including Local SGD and FedAvg,have gained much attention due to their superior properties,such as low communication cost and privacypreserving.Nevertheless,when the data distribution on workers is non-identical,local-based algorithms would encounter a significant degradation in the convergence rate.In this paper,we propose Variance Reduced Local SGD(VRL-SGD)to deal with the heterogeneous data.Without extra communication cost,VRL-SGD can reduce the gradient variance among workers caused by the heterogeneous data,and thus it prevents local-based algorithms from slow convergence rate.Moreover,we present VRL-SGD-W with an effectivewarm-up mechanism for the scenarios,where the data among workers are quite diverse.Benefiting from eliminating the impact of such heterogeneous data,we theoretically prove that VRL-SGD achieves a linear iteration speedup with lower communication complexity even if workers access non-identical datasets.We conduct experiments on three machine learning tasks.The experimental results demonstrate that VRL-SGD performs impressively better than Local SGD for the heterogeneous data and VRL-SGD-W is much robust under high data variance among workers.
文摘车联网在智慧城市建设中扮演着不可或缺的角色,汽车不仅仅是交通工具,更是大数据时代信息采集和传输的重要载体.随着车辆采集的数据量飞速增长和人们隐私保护意识的增强,如何在车联网环境中确保用户数据安全,防止数据泄露,成为亟待解决的难题.联邦学习采用“数据不动模型动”的方式,为保护用户隐私和实现良好性能提供了可行方案.然而,受限于采集设备、地域环境、个人习惯的差异,多台车辆采集的数据通常表现为非独立同分布(non-independent and identically distributed,non-IID)数据,而传统的联邦学习算法在non-IID数据环境中,其模型收敛速度较慢.针对这一挑战,提出了一种面向non-IID数据的车联网多阶段联邦学习机制,称为FedWO.第1阶段采用联邦平均算法,使得全局模型快速达到一个基本的模型准确度;第2阶段采用联邦加权多方计算,依据各车辆的数据特性计算其在全局模型中的权重,聚合后得到性能更优的全局模型,同时采用传输控制策略,减少模型传输带来的通信开销;第3阶段为个性化计算阶段,车辆利用各自的数据进行个性化学习,微调本地模型获得与本地数据更匹配的模型.实验采用了驾驶行为数据集进行实验评估,结果表明相较于传统方法,在non-IID数据场景下,FedWO机制保护了数据隐私,同时提高了算法的准确度.
文摘个性化联邦学习侧重于为各客户端提供个性化模型,旨在提高对异构数据的处理性能,然而现有的个性化联邦学习算法大多以增加客户端参数量为代价提高个性化模型的性能,使计算变得复杂.为了解决此问题,文中提出基于稀疏正则双层优化的个性化联邦学习算法(Personalized Federated Learning Based on Sparsity Regularized Bi-level Optimization,pFedSRB),在客户端的个性化更新中引入l 1范数稀疏正则化,提升个性化模型的稀疏度,避免不必要的客户端参数更新,降低模型复杂度.将个性化联邦学习建模为双层优化问题,内层优化采用交替方向乘子法,可提高学习速度.在4个联邦学习基准数据集上的实验表明,pFedSRB在异构数据上表现出色,在提高模型性能的同时有效降低训练用时和空间成本.
基金National Natural Science Foundation of China,Grant/Award Number:62272114Joint Research Fund of Guangzhou and University,Grant/Award Number:202201020380+3 种基金Guangdong Higher Education Innovation Group,Grant/Award Number:2020KCXTD007Pearl River Scholars Funding Program of Guangdong Universities(2019)National Key R&D Program of China,Grant/Award Number:2022ZD0119602Major Key Project of PCL,Grant/Award Number:PCL2022A03。
文摘As the scale of federated learning expands,solving the Non-IID data problem of federated learning has become a key challenge of interest.Most existing solutions generally aim to solve the overall performance improvement of all clients;however,the overall performance improvement often sacrifices the performance of certain clients,such as clients with less data.Ignoring fairness may greatly reduce the willingness of some clients to participate in federated learning.In order to solve the above problem,the authors propose Ada-FFL,an adaptive fairness federated aggregation learning algorithm,which can dynamically adjust the fairness coefficient according to the update of the local models,ensuring the convergence performance of the global model and the fairness between federated learning clients.By integrating coarse-grained and fine-grained equity solutions,the authors evaluate the deviation of local models by considering both global equity and individual equity,then the weight ratio will be dynamically allocated for each client based on the evaluated deviation value,which can ensure that the update differences of local models are fully considered in each round of training.Finally,by combining a regularisation term to limit the local model update to be closer to the global model,the sensitivity of the model to input perturbations can be reduced,and the generalisation ability of the global model can be improved.Through numerous experiments on several federal data sets,the authors show that our method has more advantages in convergence effect and fairness than the existing baselines.
基金supported in part by the National Natural Science Foundation of China under Grant U22B2005,Grant 62372462.
文摘The rapid expansion of artificial intelligence(AI)applications has raised significant concerns about user privacy,prompting the development of privacy-preserving machine learning(ML)paradigms such as federated learning(FL).FL enables the distributed training of ML models,keeping data on local devices and thus addressing the privacy concerns of users.However,challenges arise from the heterogeneous nature of mobile client devices,partial engagement of training,and non-independent identically distributed(non-IID)data distribution,leading to performance degradation and optimization objective bias in FL training.With the development of 5G/6G networks and the integration of cloud computing edge computing resources,globally distributed cloud computing resources can be effectively utilized to optimize the FL process.Through the specific parameters of the server through the selection mechanism,it does not increase the monetary cost and reduces the network latency overhead,but also balances the objectives of communication optimization and low engagement mitigation that cannot be achieved simultaneously in a single-server framework of existing works.In this paper,we propose the FedAdaSS algorithm,an adaptive parameter server selection mechanism designed to optimize the training efficiency in each round of FL training by selecting the most appropriate server as the parameter server.Our approach leverages the flexibility of cloud resource computing power,and allows organizers to strategically select servers for data broadcasting and aggregation,thus improving training performance while maintaining cost efficiency.The FedAdaSS algorithm estimates the utility of client systems and servers and incorporates an adaptive random reshuffling strategy that selects the optimal server in each round of the training process.Theoretical analysis confirms the convergence of FedAdaSS under strong convexity and L-smooth assumptions,and comparative experiments within the FLSim framework demonstrate a reduction in training round-to-accuracy by 12%–20%compared to the Federated Averaging(FedAvg)with random reshuffling method under unique server.Furthermore,FedAdaSS effectively mitigates performance loss caused by low client engagement,reducing the loss indicator by 50%.
