Task Offloading and Resource Allocation in NOMA-VEC:A Multi-Agent Deep Graph Reinforcement Learning Algorithm

Vehicular edge computing(VEC)is emerging as a promising solution paradigm to meet the requirements of compute-intensive applications in internet of vehicle(IoV).Non-orthogonal multiple access(NOMA)has advantages in improving spectrum efficiency and dealing with bandwidth scarcity and cost.It is an encouraging progress combining VEC and NOMA.In this paper,we jointly optimize task offloading decision and resource allocation to maximize the service utility of the NOMA-VEC system.To solve the optimization problem,we propose a multiagent deep graph reinforcement learning algorithm.The algorithm extracts the topological features and relationship information between agents from the system state as observations,outputs task offloading decision and resource allocation simultaneously with local policy network,which is updated by a local learner.Simulation results demonstrate that the proposed method achieves a 1.52%∼5.80%improvement compared with the benchmark algorithms in system service utility.

Policy Network-Based Dual-Agent Deep Reinforcement Learning for Multi-Resource Task Offloading in Multi-Access Edge Cloud Networks

The Multi-access Edge Cloud(MEC) networks extend cloud computing services and capabilities to the edge of the networks. By bringing computation and storage capabilities closer to end-users and connected devices, MEC networks can support a wide range of applications. MEC networks can also leverage various types of resources, including computation resources, network resources, radio resources,and location-based resources, to provide multidimensional resources for intelligent applications in 5/6G.However, tasks generated by users often consist of multiple subtasks that require different types of resources. It is a challenging problem to offload multiresource task requests to the edge cloud aiming at maximizing benefits due to the heterogeneity of resources provided by devices. To address this issue,we mathematically model the task requests with multiple subtasks. Then, the problem of task offloading of multi-resource task requests is proved to be NP-hard. Furthermore, we propose a novel Dual-Agent Deep Reinforcement Learning algorithm with Node First and Link features(NF_L_DA_DRL) based on the policy network, to optimize the benefits generated by offloading multi-resource task requests in MEC networks. Finally, simulation results show that the proposed algorithm can effectively improve the benefit of task offloading with higher resource utilization compared with baseline algorithms.

Deep Reinforcement Learning-Based Task Offloading and Service Migrating Policies in Service Caching-Assisted Mobile Edge Computing

Emerging mobile edge computing(MEC)is considered a feasible solution for offloading the computation-intensive request tasks generated from mobile wireless equipment(MWE)with limited computational resources and energy.Due to the homogeneity of request tasks from one MWE during a longterm time period,it is vital to predeploy the particular service cachings required by the request tasks at the MEC server.In this paper,we model a service caching-assisted MEC framework that takes into account the constraint on the number of service cachings hosted by each edge server and the migration of request tasks from the current edge server to another edge server with service caching required by tasks.Furthermore,we propose a multiagent deep reinforcement learning-based computation offloading and task migrating decision-making scheme(MBOMS)to minimize the long-term average weighted cost.The proposed MBOMS can learn the near-optimal offloading and migrating decision-making policy by centralized training and decentralized execution.Systematic and comprehensive simulation results reveal that our proposed MBOMS can converge well after training and outperforms the other five baseline algorithms.

Energy-Efficient Traffic Offloading for RSMA-Based Hybrid Satellite Terrestrial Networks with Deep Reinforcement Learning

As the demands of massive connections and vast coverage rapidly grow in the next wireless communication networks, rate splitting multiple access(RSMA) is considered to be the new promising access scheme since it can provide higher efficiency with limited spectrum resources. In this paper, combining spectrum splitting with rate splitting, we propose to allocate resources with traffic offloading in hybrid satellite terrestrial networks. A novel deep reinforcement learning method is adopted to solve this challenging non-convex problem. However, the neverending learning process could prohibit its practical implementation. Therefore, we introduce the switch mechanism to avoid unnecessary learning. Additionally, the QoS constraint in the scheme can rule out unsuccessful transmission. The simulation results validates the energy efficiency performance and the convergence speed of the proposed algorithm.

Collapse Behavior of Pipe-Framed Greenhouses with and without Reinforcement under Snow Loading:A 3-D Finite Element Analysis

The present paper first investigates the collapse behavior of a conventional pipe-framed greenhouse under snow loading based on a 3-D finite element analysis,in which both geometrical and material non-linearities are considered.Three snow load distribution patterns related to the wind-driven snow particle movement are used in the analysis.It is found that snow load distribution affects the deformation and collapse behavior of the pipe-framed greenhouse significantly.The results obtained in this study are consistent with the actual damage observed.Next,discussion is made of the effects of reinforcements by adding members to the basic frame on the strength of the whole structure,in which seven kinds of reinforcement methods are examined.A buckling analysis is also carried out.The results indicate that the most effective reinforcement method depends on the snow load distribution pattern.

Distributed Graph Database Load Balancing Method Based on Deep Reinforcement Learning

This paper focuses on the scheduling problem of workflow tasks that exhibit interdependencies.Unlike indepen-dent batch tasks,workflows typically consist of multiple subtasks with intrinsic correlations and dependencies.It necessitates the distribution of various computational tasks to appropriate computing node resources in accor-dance with task dependencies to ensure the smooth completion of the entire workflow.Workflow scheduling must consider an array of factors,including task dependencies,availability of computational resources,and the schedulability of tasks.Therefore,this paper delves into the distributed graph database workflow task scheduling problem and proposes a workflow scheduling methodology based on deep reinforcement learning(DRL).The method optimizes the maximum completion time(makespan)and response time of workflow tasks,aiming to enhance the responsiveness of workflow tasks while ensuring the minimization of the makespan.The experimental results indicate that the Q-learning Deep Reinforcement Learning(Q-DRL)algorithm markedly diminishes the makespan and refines the average response time within distributed graph database environments.In quantifying makespan,Q-DRL achieves mean reductions of 12.4%and 11.9%over established First-fit and Random scheduling strategies,respectively.Additionally,Q-DRL surpasses the performance of both DRL-Cloud and Improved Deep Q-learning Network(IDQN)algorithms,with improvements standing at 4.4%and 2.6%,respectively.With reference to average response time,the Q-DRL approach exhibits a significantly enhanced performance in the scheduling of workflow tasks,decreasing the average by 2.27%and 4.71%when compared to IDQN and DRL-Cloud,respectively.The Q-DRL algorithm also demonstrates a notable increase in the efficiency of system resource utilization,reducing the average idle rate by 5.02%and 9.30%in comparison to IDQN and DRL-Cloud,respectively.These findings support the assertion that Q-DRL not only upholds a lower average idle rate but also effectively curtails the average response time,thereby substantially improving processing efficiency and optimizing resource utilization within distributed graph database systems.

Intelligent Power Grid Load Transferring Based on Safe Action-Correction Reinforcement Learning

When a line failure occurs in a power grid, a load transfer is implemented to reconfigure the network by changingthe states of tie-switches and load demands. Computation speed is one of the major performance indicators inpower grid load transfer, as a fast load transfer model can greatly reduce the economic loss of post-fault powergrids. In this study, a reinforcement learning method is developed based on a deep deterministic policy gradient.The tedious training process of the reinforcement learning model can be conducted offline, so the model showssatisfactory performance in real-time operation, indicating that it is suitable for fast load transfer. Consideringthat the reinforcement learning model performs poorly in satisfying safety constraints, a safe action-correctionframework is proposed to modify the learning model. In the framework, the action of load shedding is correctedaccording to sensitivity analysis results under a small discrete increment so as to match the constraints of line flowlimits. The results of case studies indicate that the proposed method is practical for fast and safe power grid loadtransfer.

Reinforcing effects of polypropylene on energy absorption and fracturing of cement-based tailings backfill under impact loading

Polypropylene(PP)fiber-reinforced cement-based tailings backfill(FRCTB)is a green compound material with superior crack resistance and has good prospects for application in underground mining.However,FRCTB exhibits susceptibility to dynamic events,such as impact ground pressure and blast vibrations.This paper investigates the energy and crack distribution behavior of FRCTB under dynamic impact,considering the height/diameter(H/D)effect.Split Hopkinson pressure bar,industrial computed tomography scan,and scanning electron microscopy(SEM)experiments were carried out on six types of FRCTB.Laboratory outcomes confirmed fiber aggregation at the bottom of specimens.When H/D was less than 0.8,the proportion of PP fibers distributed along theθangle direction of80°-90°increased.For the total energy,all samples presented similar energy absorption,reflectance,and transmittance.However,a rise in H/D may cause a rise in the energy absorption rate of FRCTB during the peak phase.A positive correlation existed between the average strain rate and absorbed energy per unit volume.The increase in H/D resulted in a decreased crack volume fraction of FRCTB.When the H/D was greater than or equal to 0.7,the maximum crack volume fraction of FRCTB was observed close to the incidence plane.Radial cracks were present only in the FRCTB with an H/D ratio of 0.5.Samples with H/D ratios of 0.5 and 0.6 showed similar distributions of weakly and heavily damaged areas.PP fibers can limit the emergence and expansion of cracks by influencing their path.SEM observations revealed considerable differences in the bonding strengths between fibers and the FRCTB.Fibers that adhered particularly well to the substrate were attracted together with the hydration products adhering to surfaces.These results show that FRCTB is promising as a sustainable and green backfill for determining the design properties of mining with backfill.

Task offloading mechanism based on federated reinforcement learning in mobile edge computing

With the arrival of 5G,latency-sensitive applications are becoming increasingly diverse.Mobile Edge Computing(MEC)technology has the characteristics of high bandwidth,low latency and low energy consumption,and has attracted much attention among researchers.To improve the Quality of Service(QoS),this study focuses on computation offloading in MEC.We consider the QoS from the perspective of computational cost,dimensional disaster,user privacy and catastrophic forgetting of new users.The QoS model is established based on the delay and energy consumption and is based on DDQN and a Federated Learning(FL)adaptive task offloading algorithm in MEC.The proposed algorithm combines the QoS model and deep reinforcement learning algorithm to obtain an optimal offloading policy according to the local link and node state information in the channel coherence time to address the problem of time-varying transmission channels and reduce the computing energy consumption and task processing delay.To solve the problems of privacy and catastrophic forgetting,we use FL to make distributed use of multiple users’data to obtain the decision model,protect data privacy and improve the model universality.In the process of FL iteration,the communication delay of individual devices is too large,which affects the overall delay cost.Therefore,we adopt a communication delay optimization algorithm based on the unary outlier detection mechanism to reduce the communication delay of FL.The simulation results indicate that compared with existing schemes,the proposed method significantly reduces the computation cost on a device and improves the QoS when handling complex tasks.

Dynamic Task Offloading for Digital Twin-Empowered Mobile Edge Computing via Deep Reinforcement Learning

Limited by battery and computing re-sources,the computing-intensive tasks generated by Internet of Things(IoT)devices cannot be processed all by themselves.Mobile edge computing(MEC)is a suitable solution for this problem,and the gener-ated tasks can be offloaded from IoT devices to MEC.In this paper,we study the problem of dynamic task offloading for digital twin-empowered MEC.Digital twin techniques are applied to provide information of environment and share the training data of agent de-ployed on IoT devices.We formulate the task offload-ing problem with the goal of maximizing the energy efficiency and the workload balance among the ESs.Then,we reformulate the problem as an MDP problem and design DRL-based energy efficient task offloading(DEETO)algorithm to solve it.Comparative experi-ments are carried out which show the superiority of our DEETO algorithm in improving energy efficiency and balancing the workload.

Multi-Agent Deep Reinforcement Learning for Efficient Computation Offloading in Mobile Edge Computing

Mobile-edge computing(MEC)is a promising technology for the fifth-generation(5G)and sixth-generation(6G)architectures,which provides resourceful computing capabilities for Internet of Things(IoT)devices,such as virtual reality,mobile devices,and smart cities.In general,these IoT applications always bring higher energy consumption than traditional applications,which are usually energy-constrained.To provide persistent energy,many references have studied the offloading problem to save energy consumption.However,the dynamic environment dramatically increases the optimization difficulty of the offloading decision.In this paper,we aim to minimize the energy consumption of the entireMECsystemunder the latency constraint by fully considering the dynamic environment.UnderMarkov games,we propose amulti-agent deep reinforcement learning approach based on the bi-level actorcritic learning structure to jointly optimize the offloading decision and resource allocation,which can solve the combinatorial optimization problem using an asymmetric method and compute the Stackelberg equilibrium as a better convergence point than Nash equilibrium in terms of Pareto superiority.Our method can better adapt to a dynamic environment during the data transmission than the single-agent strategy and can effectively tackle the coordination problem in the multi-agent environment.The simulation results show that the proposed method could decrease the total computational overhead by 17.8%compared to the actor-critic-based method and reduce the total computational overhead by 31.3%,36.5%,and 44.7%compared with randomoffloading,all local execution,and all offloading execution,respectively.

UAV-Assisted Dynamic Avatar Task Migration for Vehicular Metaverse Services: A Multi-Agent Deep Reinforcement Learning Approach

Avatars, as promising digital representations and service assistants of users in Metaverses, can enable drivers and passengers to immerse themselves in 3D virtual services and spaces of UAV-assisted vehicular Metaverses. However, avatar tasks include a multitude of human-to-avatar and avatar-to-avatar interactive applications, e.g., augmented reality navigation,which consumes intensive computing resources. It is inefficient and impractical for vehicles to process avatar tasks locally. Fortunately, migrating avatar tasks to the nearest roadside units(RSU)or unmanned aerial vehicles(UAV) for execution is a promising solution to decrease computation overhead and reduce task processing latency, while the high mobility of vehicles brings challenges for vehicles to independently perform avatar migration decisions depending on current and future vehicle status. To address these challenges, in this paper, we propose a novel avatar task migration system based on multi-agent deep reinforcement learning(MADRL) to execute immersive vehicular avatar tasks dynamically. Specifically, we first formulate the problem of avatar task migration from vehicles to RSUs/UAVs as a partially observable Markov decision process that can be solved by MADRL algorithms. We then design the multi-agent proximal policy optimization(MAPPO) approach as the MADRL algorithm for the avatar task migration problem. To overcome slow convergence resulting from the curse of dimensionality and non-stationary issues caused by shared parameters in MAPPO, we further propose a transformer-based MAPPO approach via sequential decision-making models for the efficient representation of relationships among agents. Finally, to motivate terrestrial or non-terrestrial edge servers(e.g., RSUs or UAVs) to share computation resources and ensure traceability of the sharing records, we apply smart contracts and blockchain technologies to achieve secure sharing management. Numerical results demonstrate that the proposed approach outperforms the MAPPO approach by around 2% and effectively reduces approximately 20% of the latency of avatar task execution in UAV-assisted vehicular Metaverses.

Cognitive interference decision method for air defense missile fuze based on reinforcement learning

To solve the problem of the low interference success rate of air defense missile radio fuzes due to the unified interference form of the traditional fuze interference system,an interference decision method based Q-learning algorithm is proposed.First,dividing the distance between the missile and the target into multiple states to increase the quantity of state spaces.Second,a multidimensional motion space is utilized,and the search range of which changes with the distance of the projectile,to select parameters and minimize the amount of ineffective interference parameters.The interference effect is determined by detecting whether the fuze signal disappears.Finally,a weighted reward function is used to determine the reward value based on the range state,output power,and parameter quantity information of the interference form.The effectiveness of the proposed method in selecting the range of motion space parameters and designing the discrimination degree of the reward function has been verified through offline experiments involving full-range missile rendezvous.The optimal interference form for each distance state has been obtained.Compared with the single-interference decision method,the proposed decision method can effectively improve the success rate of interference.

Damage evolution of rock-encased-backfill structure under stepwise cyclic triaxial loading

Rock-encased-backfill(RB)structures are common in underground mining,for example in the cut-andfill and stoping methods.To understand the effects of cyclic excavation and blasting activities on the damage of these RB structures,a series of triaxial stepwise-increasing-amplitude cyclic loading experiments was conducted with cylindrical RB specimens(rock on outside,backfill on inside)with different volume fractions of rock(VF=0.48,0.61,0.73,and 0.84),confining pressures(0,6,9,and 12 MPa),and cyclic loading rates(200,300,400,and 500 N/s).The damage evolution and meso-crack formation during the cyclic tests were analyzed with results from stress-strain hysteresis loops,acoustic emission events,and post-failure X-ray 3D fracture morphology.The results showed significant differences between cyclic and monotonic loadings of RB specimens,particularly with regard to the generation of shear microcracks,the development of stress memory and strain hardening,and the contact forces and associated friction that develops along the rock-backfill interface.One important finding is that as a function of the number of cycles,the elastic strain increases linearly and the dissipated energy increases exponentially.Also,compared with monotonic loading,the cyclic strain hardening characteristics are more sensitive to rising confining pressures during the initial compaction stage.Another finding is that compared with monotonic loading,more shear microcracks are generated during every reloading stage,but these microcracks tend to be dispersed and lessen the likelihood of large shear fracture formation.The transition from elastic to plastic behavior varies depending on the parameters of each test(confinement,volume fraction,and cyclic rate),and an interesting finding was that the transformation to plastic behavior is significantly lower under the conditions of 0.73 rock volume fraction,400 N/s cyclic loading rate,and 9 MPa confinement.All the findings have important practical implications on the ability of backfill to support underground excavations.

Liquefaction susceptibility and deformation characteristics of saturated coral sandy soils subjected to cyclic loadings-a critical review

Coral sandy soils widely exist in coral island reefs and seashores in tropical and subtropical regions.Due to the unique marine depositional environment of coral sandy soils,the engineering characteristics and responses of these soils subjected to monotonic and cyclic loadings have been a subject of intense interest among the geotechnical and earthquake engineering communities.This paper critically reviews the progress of experimental investigations on the undrained behavior of coral sandy soils under monotonic and cyclic loadings over the last three decades.The focus of coverage includes the contractive-dilative behavior,the pattern of excess pore-water pressure(EPWP)generation and the liquefaction mechanism and liquefaction resistance,the small-strain shear modulus and strain-dependent shear modulus and damping,the cyclic softening feature,and the anisotropic characteristics of undrained responses of saturated coral sandy soils.In particular,the advances made in the past decades are reviewed from the following aspects:(1)the characterization of factors that impact the mechanism and patterns of EPWP build-up;(2)the identification of liquefaction triggering in terms of the apparent viscosity and the average flow coefficient;(3)the establishment of the invariable form of strain-based,stress-based,or energy-based EPWP ratio formulas and the unique relationship between the new proxy of liquefaction resistance and the number of cycles required to reach liquefaction;(4)the establishment of the invariable form of the predictive formulas of small strain modulus and strain-dependent shear modulus;and(5)the investigation on the effects of stress-induced anisotropy on liquefaction susceptibility and dynamic deformation characteristics.Insights gained through the critical review of these advances in the past decades offer a perspective for future research to further resolve the fundamental issues concerning the liquefaction mechanism and responses of coral sandy sites subjected to cyclic loadings associated with seismic events in marine environments.

Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Lateral earth pressure of granular backfills on retaining walls with expanded polystyrene geofoam inclusions under limited surcharge loading

Existing studies have focused on the behavior of the retaining wall equipped with expanded polystyrene(EPS)geofoam inclusions under semi-infinite surcharge loading rather than limited surcharge loading.In this paper,the failure mode and the earth pressure acting on the rigid retaining wall with EPS geofoam inclusions and granular backfills(henceforth referred to as EPS-wall),under limited surcharge loading are investigated through two-and three-dimensional model tests.The testing results show that different from the sliding of almost all the backfill in the EPS-wall under semi-infinite surcharge loading,only an approximately triangular backfill slides in the wall under limited surcharge loading.The distribution of the lateral earth pressure on the EPS-wall under limited surcharge loading is non-linear,and the distribution changes from the increase of the wall depth to the decrease with the increase of the limited surcharge loading.An approach based on the force equilibrium of a differential element is developed to predict the lateral earth pressure behind the EPS-wall subjected to limited surcharge loading,and its performance was fully validated by the three-dimensional model tests.

Dynamic Economic Scheduling with Self-Adaptive Uncertainty in Distribution Network Based on Deep Reinforcement Learning

Traditional optimal scheduling methods are limited to accurate physical models and parameter settings, which aredifficult to adapt to the uncertainty of source and load, and there are problems such as the inability to make dynamicdecisions continuously. This paper proposed a dynamic economic scheduling method for distribution networksbased on deep reinforcement learning. Firstly, the economic scheduling model of the new energy distributionnetwork is established considering the action characteristics of micro-gas turbines, and the dynamic schedulingmodel based on deep reinforcement learning is constructed for the new energy distribution network system with ahigh proportion of new energy, and the Markov decision process of the model is defined. Secondly, Second, for thechanging characteristics of source-load uncertainty, agents are trained interactively with the distributed networkin a data-driven manner. Then, through the proximal policy optimization algorithm, agents adaptively learn thescheduling strategy and realize the dynamic scheduling decision of the new energy distribution network system.Finally, the feasibility and superiority of the proposed method are verified by an improved IEEE 33-node simulationsystem.

Mechanical responses of anchoring structure under triaxial cyclic loading

Dynamic load on anchoring structures(AS)within deep roadways can result in cumulative damage and failure.This study develops an experimental device designed to test AS under triaxial loads.The device enables the investigation of the mechanical response,failure mode,instability assessment criteria,and anchorage effect of AS subjected to combined cyclic dynamic-static triaxial stress paths.The results show that the peak bearing strength is positively correlated with the anchoring matrix strength,anchorage length,and edgewise compressive strength.The bearing capacity decreases significantly when the anchorage direction is severely inclined.The free face failure modes are typically transverse cracking,concave fracturing,V-shaped slipping and detachment,and spallation detachment.Besides,when the anchoring matrix strength and the anchorage length decrease while the edgewise compressive strength,loading rate,and anchorage inclination angle increase,the failure intensity rises.Instability is determined by a negative tangent modulus of the displacement-strength curve or the continued deformation increase against the general downward trend.Under cyclic loads,the driving force that breaks the rock mass along the normal vector and the rigidity of the AS are the two factors that determine roadway stability.Finally,a control measure for surrounding rock stability is proposed to reduce the internal driving force via a pressure relief method and improve the rigidity of the AS by full-length anchorage and grouting modification.

Commentary on"Speed and surface steepness affect internal tibial loading during running"

It is a pleasure to write a commentary on the work of Dr.Hannah Rice and colleagues,l who have advanced our understanding of how the mechanical loading environment of the tibia changes as a function of running grade and speed.It is important that we understand how the tibia is loaded during conditions that an individual is likely to encounter when running as it is these internal loads which we believe are responsible for the development of bone-stress injuries.