A Comprehensive Survey on Joint Resource Allocation Strategies in Federated Edge Learning

Federated Edge Learning(FEL),an emerging distributed Machine Learning(ML)paradigm,enables model training in a distributed environment while ensuring user privacy by using physical separation for each user’s data.However,with the development of complex application scenarios such as the Internet of Things(IoT)and Smart Earth,the conventional resource allocation schemes can no longer effectively support these growing computational and communication demands.Therefore,joint resource optimization may be the key solution to the scaling problem.This paper simultaneously addresses the multifaceted challenges of computation and communication,with the growing multiple resource demands.We systematically review the joint allocation strategies for different resources(computation,data,communication,and network topology)in FEL,and summarize the advantages in improving system efficiency,reducing latency,enhancing resource utilization,and enhancing robustness.In addition,we present the potential ability of joint optimization to enhance privacy preservation by reducing communication requirements,indirectly.This work not only provides theoretical support for resource management in federated learning(FL)systems,but also provides ideas for potential optimal deployment in multiple real-world scenarios.By thoroughly discussing the current challenges and future research directions,it also provides some important insights into multi-resource optimization in complex application environments.

A Fully-Integrated Memristor Chip for Edge Learning

It is still challenging to fully integrate computing in memory chip as edge learning devices.In recent work published on Science,a fully-integrated chip based on neuromorphic memristors was developed for edge learning as artificial neural networks with functionality of synapses,dendrites,and somas.A crossbar-array memristor chip facilitated edge learning including hardware realization,learning algorithm,and cycle-parallel sign-and threshold-based learning(STELLAR)scheme.The motion control and demonstration platforms were executed to improve the edge learning ability for adapting to new scenarios.

Adaptive Retransmission Design for Wireless Federated Edge Learning

As a popular distributed machine learning framework,wireless federated edge learning(FEEL)can keep original data local,while uploading model training updates to protect privacy and prevent data silos.However,since wireless channels are usually unreliable,there is no guarantee that the model updates uploaded by local devices are correct,thus greatly degrading the performance of the wireless FEEL.Conventional retransmission schemes designed for wireless systems generally aim to maximize the system throughput or minimize the packet error rate,which is not suitable for the FEEL system.A novel retransmission scheme is proposed for the FEEL system to make a tradeoff between model training accuracy and retransmission latency.In the proposed scheme,a retransmission device selection criterion is first designed based on the channel condition,the number of local data,and the importance of model updates.In addition,we design the air interface signaling under this retransmission scheme to facilitate the implementation of the proposed scheme in practical scenarios.Finally,the effectiveness of the proposed retransmission scheme is validated through simulation experiments.

Federated Edge Learning for the Wireless Physical Layer:Opportunities and Challenges

Deep learning(DL)has been applied to the physical layer of wireless communication systems,which directly extracts environment knowledge from data and outperforms conventional methods either in accuracy or computation complexity.However,most related research works employ centralized training that inevitably involves collecting training data from edge devices.The data uploading process usually results in excessive communication overhead and privacy disclosure.Alternatively,a distributed learning approach named federated edge learning(FEEL)is introduced to physical layer designs.In FEEL,all devices collaborate to train a global model only by exchanging parameters with a nearby access point.Because all datasets are kept local,data privacy is better protected and data transmission overhead can be reduced.This paper reviews the studies on applying FEEL to the wireless physical layer including channel state information acquisition,transmitter,and receiver design,which represent a paradigm shift of the DL-based physical layer design.In the meantime they also reveal several limitations inherent in FEEL,particularly when applied to the wireless physical layer,thus motivating further research efforts in the field.

Joint User Selection and Resource Allocation for Fast Federated Edge Learning

By periodically aggregating local learning updates from edge users, federated edge learning (FEEL) is envisioned as a promising means to reap the benefit of local rich da?ta and protect users'privacy. However, the scarce wireless communication resource greatly limits the number of participated users and is regarded as the main bottleneck which hin?ders the development of FEEL. To tackle this issue, we propose a user selection policy based on data importance for FEEL system. In order to quantify the data importance of each user, we first analyze the relationship between the loss decay and the squared norm of gradi?ent. Then, we formulate a combinatorial optimization problem to maximize the learning effi?ciency by jointly considering user selection and communication resource allocation. By problem transformation and relaxation, the optimal user selection policy and resource alloca?tion are derived, and a polynomial-time optimal algorithm is developed. Finally, we deploy two commonly used deep neural network (DNN) models for simulation. The results validate that our proposed algorithm has strong generalization ability and can attain higher learning efficiency compared with other traditional algorithms.

Energy-Efficient Federated Edge Learning with Joint Communication and Computation Design

This paper studies a federated edge learning system,in which an edge server coordinates a set of edge devices to train a shared machine learning(ML)model based on their locally distributed data samples.During the distributed training,we exploit the joint communication and computation design for improving the system energy efficiency,in which both the communication resource allocation for global ML-parameters aggregation and the computation resource allocation for locally updating ML-parameters are jointly optimized.In particular,we consider two transmission protocols for edge devices to upload ML-parameters to edge server,based on the non-orthogonal multiple access(NOMA)and time division multiple access(TDMA),respectively.Under both protocols,we minimize the total energy consumption at all edge devices over a particular finite training duration subject to a given training accuracy,by jointly optimizing the transmission power and rates at edge devices for uploading ML-parameters and their central processing unit(CPU)frequencies for local update.We propose efficient algorithms to solve the formulated energy minimization problems by using the techniques from convex optimization.Numerical results show that as compared to other benchmark schemes,our proposed joint communication and computation design significantly can improve the energy efficiency of the federated edge learning system,by properly balancing the energy tradeoff between communication and computation.

Training time minimization for federated edge learning with optimized gradient quantization and bandwidth allocation

Training a machine learning model with federated edge learning(FEEL)is typically time consuming due to the constrained computation power of edge devices and the limited wireless resources in edge networks.In this study,the training time minimization problem is investigated in a quantized FEEL system,where heterogeneous edge devices send quantized gradients to the edge server via orthogonal channels.In particular,a stochastic quantization scheme is adopted for compression of uploaded gradients,which can reduce the burden of per-round communication but may come at the cost of increasing the number of communication rounds.The training time is modeled by taking into account the communication time,computation time,and the number of communication rounds.Based on the proposed training time model,the intrinsic trade-off between the number of communication rounds and per-round latency is characterized.Specifically,we analyze the convergence behavior of the quantized FEEL in terms of the optimality gap.Furthermore,a joint data-and-model-driven fitting method is proposed to obtain the exact optimality gap,based on which the closed-form expressions for the number of communication rounds and the total training time are obtained.Constrained by the total bandwidth,the training time minimization problem is formulated as a joint quantization level and bandwidth allocation optimization problem.To this end,an algorithm based on alternating optimization is proposed,which alternatively solves the subproblem of quantization optimization through successive convex approximation and the subproblem of bandwidth allocation by bisection search.With different learning tasks and models,the validation of our analysis and the near-optimal performance of the proposed optimization algorithm are demonstrated by the simulation results.

Distributionally Robust Learning Based on Dirichlet Process Prior in Edge Networks

In order to meet the real-time performance requirements,intelligent decisions in Internet of things applications must take place right here right now at the network edge.Pushing the artificial intelligence frontier to achieve edge intelligence is nontrivial due to the constrained computing resources and limited training data at the network edge.To tackle these challenges,we develop a distributionally robust optimization(DRO)-based edge learning algorithm,where the uncertainty model is constructed to foster the synergy of cloud knowledge and local training.Specifically,the cloud transferred knowledge is in the form of a Dirichlet process prior distribution for the edge model parameters,and the edge device further constructs an uncertainty set centered around the empirical distribution of its local samples.The edge learning DRO problem,subject to these two distributional uncertainty constraints,is recast as a single-layer optimization problem using a duality approach.We then use an Expectation-Maximization algorithm-inspired method to derive a convex relaxation,based on which we devise algorithms to learn the edge model.Furthermore,we illustrate that the meta-learning fast adaptation procedure is equivalent to our proposed Dirichlet process prior-based approach.Finally,extensive experiments are implemented to showcase the performance gain over standard approaches using edge data only.

MobileNet network optimization based on convolutional block attention module

Deep learning technology is widely used in computer vision.Generally,a large amount of data is used to train the model weights in deep learning,so as to obtain a model with higher accuracy.However,massive data and complex model structures require more calculating resources.Since people generally can only carry and use mobile and portable devices in application scenarios,neural networks have limitations in terms of calculating resources,size and power consumption.Therefore,the efficient lightweight model MobileNet is used as the basic network in this study for optimization.First,the accuracy of the MobileNet model is improved by adding methods such as the convolutional block attention module(CBAM)and expansion convolution.Then,the MobileNet model is compressed by using pruning and weight quantization algorithms based on weight size.Afterwards,methods such as Python crawlers and data augmentation are employed to create a garbage classification data set.Based on the above model optimization strategy,the garbage classification mobile terminal application is deployed on mobile phones and raspberry pies,realizing completing the garbage classification task more conveniently.

An Overview of Data-Importance Aware Radio Resource Management for Edge Machine Learning

The 5G network connecting billions of Internet of things(IoT)devices will make it possible to harvest an enormous amount of real-time mobile data.Furthermore,the 5G virtualization architecture will enable cloud computing at the(network)edge.The availability of both rich data and computation power at the edge has motivated Internet companies to deploy artificial intelligence(AI)there,creating the hot area of edge-AI.Edge learning,the theme of this project,concerns training edge-AI models,which endow on IoT devices intelligence for responding to real-time events.However,the transmission of high-dimensional data from many edge devices to servers can result in excessive communication latency,creating a bottleneck for edge learning.Traditional wireless techniques deigned for only radio access are ineffective in tackling the challenge.Attempts to overcome the communication bottleneck has led to the development of a new class of techniques for intelligent radio resource management(RRM),called data-importance aware RRM.Their designs feature the interplay of active machine learning and wireless communication.Specifically,the metrics that measure data importance in active learning(e.g.,classification uncertainty and data diversity)are applied to RRM for efficient acquisition of distributed data in wireless networks to train AI models at servers.This article aims at providing an introduction to the emerging area of importance-aware RRM.To this end,we will introduce the design principles,survey recent advancements in the area,discuss some design examples,and suggest some promising research opportunities.