Stochastic Learning for Opportunistic Peer-to-Peer Computation Offloading in IoT Edge Computing

Peer-to-peer computation offloading has been a promising approach that enables resourcelimited Internet of Things(IoT)devices to offload their computation-intensive tasks to idle peer devices in proximity.Different from dedicated servers,the spare computation resources offered by peer devices are random and intermittent,which affects the offloading performance.The mutual interference caused by multiple simultaneous offloading requestors that share the same wireless channel further complicates the offloading decisions.In this work,we investigate the opportunistic peer-to-peer task offloading problem by jointly considering the stochastic task arrivals,dynamic interuser interference,and opportunistic availability of peer devices.Each requestor makes decisions on both local computation frequency and offloading transmission power to minimize its own expected long-term cost on tasks completion,which takes into consideration its energy consumption,task delay,and task loss due to buffer overflow.The dynamic decision process among multiple requestors is formulated as a stochastic game.By constructing the post-decision states,a decentralized online offloading algorithm is proposed,where each requestor as an independent learning agent learns to approach the optimal strategies with its local observations.Simulation results under different system parameter configurations demonstrate the proposed online algorithm achieves a better performance compared with some existing algorithms,especially in the scenarios with large task arrival probability or small helper availability probability.

A MAXIMUM PRINCIPLE APPROACH TO STOCHASTIC H_2/H_∞ CONTROL WITH RANDOM JUMPS

A necessary maximum principle is given for nonzero-sum stochastic Oltterential games with random jumps. The result is applied to solve the H2/H∞ control problem of stochastic systems with random jumps. A necessary and sufficient condition for the existence of a unique solution to the H2/H∞ control problem is derived. The resulting solution is given by the solution of an uncontrolled forward backward stochastic differential equation with random jumps.

Open Loop Saddle Point on Linear Quadratic Stochastic Differential Games

In this paper, we deal with one kind of two-player zero-sum linear quadratic stochastic differential game problem. We give the existence of an open loop saddle point if and only if the lower and upper values exist.

Optimal Strategies Trajectory with Multi-Local-Worlds Graph

This paper constructs a non-cooperative/cooperative stochasticdifferential game model to prove that the optimal strategies trajectory ofagents in a system with a topological configuration of a Multi-Local-Worldgraph would converge into a certain attractor if the system’s configuration isfixed. Due to the economics and management property, almost all systems aredivided into several independent Local-Worlds, and the interaction betweenagents in the system is more complex. The interaction between agents inthe same Local-World is defined as a stochastic differential cooperativegame;conversely, the interaction between agents in different Local-Worldsis defined as a stochastic differential non-cooperative game. We construct anon-cooperative/cooperative stochastic differential game model to describethe interaction between agents. The solutions of the cooperative and noncooperativegames are obtained by invoking corresponding theories, and thena nonlinear operator is constructed to couple these two solutions together.At last, the optimal strategies trajectory of agents in the system is proven toconverge into a certain attractor, which means that strategies trajectory arecertainty as time tends to infinity or a large positive integer. It is concluded thatthe optimal strategy trajectory with a nonlinear operator of cooperative/noncooperativestochastic differential game between agents can make agentsin a certain Local-World coordinate and make the Local-World paymentmaximize, and can make the all Local-Worlds equilibrated;furthermore, theoptimal strategy of the coupled game can converge into a particular attractorthat decides the optimal property.

Strong Subgame Consistency of the Core in Stochastic Network Formation Games

We consider a model of network formation as a stochastic game with random duration proposed initially in Sun and Parilina(Autom Remote Control 82(6):1065–1082,2021).In the model,the leader first suggests a joint project to other players,i.e.,the network connecting them.Second,the players are allowed to form fresh links with each other updating the initially proposed network.The stage payoff of any player is defined depending on the network structure.There are two types of randomness in the network formation process:(i)links may fail to be formed with different probabilities although players intend to establish them,(ii)the game process may terminate at any stage or transit to the next stage with a certain probability distribution.Finally,a network is formed as a result of players’decisions and realization of random variables.The cooperative version of the stochastic game is investigated.In particular,we examine the properties of subgame consistency as well as strong subgame consistency of the core.We provide a payment mechanism or regularization of the core elements to sustain its subgame consistency and avoid the player’s deviations from the cooperative trajectory.In addition,the distribution procedure of the core elements is regularized in case there are negative payments to achieve only nonnegative payments to the players at any stage.The sufficient condition of a strongly subgame consistent core is also obtained.We illustrate our theoretical results with a numerical example.

LINEAR QUADRATIC NONZERO-SUM DIFFERENTIAL GAMES WITH RANDOM JUMPS

The existence and uniqueness of the solutions for one kind of forward- backward stochastic differential equations with Brownian motion and Poisson process as the noise source were given under the monotone conditions. Then these results were applied to nonzero-sum differential games with random jumps to get the explicit form of the open-loop Nash equilibrium point by the solution of the forward-backward stochastic differential equations.

Efficient Deployment of Multiple UAVs for IoT Communication in Dynamic Environment

The application of unmanned aerial vehicle(UAV)-mounted base stations is emerging as an effective solution to provide wireless communication service for a target region containing some smart objects(SOs)in internet of things(IoT).This paper investigates the efficient deployment problem of multiple UAVs for IoT communication in dynamic environment.We first define a measurement of communication performance of UAVto-SO in the target region which is regarded as the optimization objective.The state of one SO is active when it needs to transmit or receive the data;otherwise,silent.The switch of two different states is implemented with a certain probability that results in a dynamic communication environment.In the dynamic environment,the active states of SOs cannot be known by UAVs in advance and only neighbouring UAVs can communicate with each other.To overcome these challenges in the deployment,we leverage a game-theoretic learning approach to solve the position-selected problem.This problem is modeled a stochastic game,which is proven that it is an exact potential game and exists the best Nash equilibria(NE).Furthermore,a distributed position optimization algorithm is proposed,which can converge to a pure-strategy NE.Numerical results demonstrate the excellent performance of our proposed algorithm.

Linear Quadratic Leader-Follower Stochastic Differential Games:Closed-Loop Solvability

In this paper,a leader-follower stochastic differential game is studied for a linear stochastic differential equation with quadratic cost functionals.The coefficients in the state equation and the weighting matrices in the cost functionals are all deterministic.Closed-loop strategies are introduced,which require to be independent of initial states;and such a nature makes it very useful and convenient in applications.The follower first solves a stochastic linear quadratic optimal control problem,and his optimal closed-loop strategy is characterized by a Riccati equation,together with an adapted solution to a linear backward stochastic differential equation.Then the leader turns to solve a stochastic linear quadratic optimal control problem of a forward-backward stochastic differential equation,necessary conditions for the existence of the optimal closed-loop strategy for the leader is given by a Riccati equation.Some examples are also given.

Zero-Sum Stochastic Games with Average Payoffs:New Optimality Conditions

In this paper we study zero-sum stochastic games. The optimality criterion is the long-run expected average criterion, and the payoff function may have neither upper nor lower bounds. We give a new set of conditions for the existence of a value and a pair of optimal stationary strategies. Our conditions are slightly weaker than those in the previous literature, and some new sufficient conditions for the existence of a pair of optimal stationary strategies are imposed on the primitive data of the model. Our results are illustrated with a queueing system, for which our conditions are satisfied but some of the conditions in some previous literatures fail to hold.

Stochastic Nash Games for Markov Jump Linear Systems with State-and Control-Dependent Noise

This paper investigates Nash games for a class of linear stochastic systems governed by Itô’s differential equation with Markovian jump parameters both in finite-time horizon and infinite-time horizon.First,stochastic Nash games are formulated by applying the results of indefinite stochastic linear quadratic(LQ)control problems.Second,in order to obtain Nash equilibrium strategies,crosscoupled stochastic Riccati differential(algebraic)equations(CSRDEs and CSRAEs)are derived.Moreover,in order to demonstrate the validity of the obtained results,stochastic H2/H∞control with state-and control-dependent noise is discussed as an immediate application.Finally,a numerical example is provided.

Dynamic computation offloading in time-varying environment for ultra-dense networks:a stochastic game approach

To meet the demands of large-scale user access with computation-intensive and delay-sensitive applications,combining ultra-dense networks(UDNs)and mobile edge computing(MEC)are considered as important solutions.In the MEC enabled UDNs,one of the most important issues is computation offloading.Although a number of work have been done toward this issue,the problem of dynamic computation offloading in time-varying environment,especially the dynamic computation offloading problem for multi-user,has not been fully considered.Therefore,in order to fill this gap,the dynamic computation offloading problem in time-varying environment for multi-user is considered in this paper.By considering the dynamic changes of channel state and users’queue state,the dynamic computation offloading problem for multi-user is formulated as a stochastic game,which aims to optimize the delay and packet loss rate of users.To find the optimal solution of the formulated optimization problem,Nash Q-learning(NQLN)algorithm is proposed which can be quickly converged to a Nash equilibrium solution.Finally,extensive simulation results are presented to demonstrate the superiority of NQLN algorithm.It is shown that NQLN algorithm has better optimization performance than the benchmark schemes.

Stochastic Differential Games with Reflection and Related Obstacle Problems for Isaacs Equations

In this paper we first investigate zero-sum two-player stochastic differential games with reflection, with the help of theory of Reflected Backward Stochastic Differential Equations （RBSDEs）. We will establish the dynamic programming principle for the upper and the lower value functions of this kind of stochastic differential games with reflection in a straightforward way. Then the upper and the lower value functions are proved to be the unique viscosity solutions to the associated upper and the lower Hamilton-Jacobi-Bettman-Isaacs equations with obstacles, respectively. The method differs significantly from those used for control problems with reflection, with new techniques developed of interest on its own. Further, we also prove a new estimate for RBSDEs being sharper than that in the paper of E1 Karoui, Kapoudjian, Pardoux, Peng and Quenez （1997）, which turns out to be very useful because it allows us to estimate the LP-distance of the solutions of two different RBSDEs by the p-th power of the distance of the initial values of the driving forward equations. We also show that the unique viscosity solution to the approximating Isaacs equation constructed by the penalization method converges to the viscosity solution of the Isaacs equation with obstacle.

Two-Stage Stochastic Variational Inequalities: Theory, Algorithms and Applications

The stochastic variational inequality(SVI)provides a unified form of optimality con-ditions of stochastic optimization and stochastic games which have wide applications in science,engineering,economics and finance.In the recent two decades,one-stage SVI has been studied extensively and widely used in modeling equilibrium problems under uncertainty.Moreover,the recently proposed two-stage SVI and multistage SVI can be applied to the case when the decision makers want to make decisions at different stages in a stochastic environment.The two-stage SVI is a foundation of multistage SVI,which is to find a pair of“here-and-now”solution and“wait-and-see”solution.This paper provides a survey of recent developments in analysis,algorithms and applications of the two-stage SVI.

Maximum Principle for Non-Zero Sum Stochastic Differential Game with Discrete and Distributed Delays

This technical note is concerned with the maximum principle for a non-zero sum stochastic differential game with discrete and distributed delays.Not only the state variable,but also control variables of players involve discrete and distributed delays.By virtue of the duality method and the generalized anticipated backward stochastic differential equations,the author establishes a necessary maximum principle and a sufficient verification theorem.To explain theoretical results,the author applies them to a dynamic advertising game problem.

A BSDE Approach to Stochastic Differential Games Involving Impulse Controls and HJBI Equation

This paper focuses on zero-sum stochastic differential games in the framework of forwardbackward stochastic differential equations on a finite time horizon with both players adopting impulse controls.By means of BSDE methods,in particular that of the notion from Peng’s stochastic backward semigroups,the authors prove a dynamic programming principle for both the upper and the lower value functions of the game.The upper and the lower value functions are then shown to be the unique viscosity solutions of the Hamilton-Jacobi-Bellman-Isaacs equations with a double-obstacle.As a consequence,the uniqueness implies that the upper and lower value functions coincide and the game admits a value.

Mean-field type forward-backward doubly stochastic differential equations and related stochastic differential games

We study a kind of partial information non-zero sum differential games of mean-field backward doubly stochastic differential equations,in which the coefficient contains not only the state process but also its marginal distribution,and the cost functional is also of mean-field type.It is required that the control is adapted to a sub-filtration of the filtration generated by the underlying Brownian motions.We establish a necessary condition in the form of maximum principle and a verification theorem,which is a sufficient condition for Nash equilibrium point.We use the theoretical results to deal with a partial information linear-quadratic(LQ)game,and obtain the unique Nash equilibrium point for our LQ game problem by virtue of the unique solvability of mean-field forward-backward doubly stochastic differential equation.

Policy Iteration Algorithms for Zero-Sum Stochastic Differential Games with Long-Run Average Payoff Criteria

This paper studies the policy iteration algorithm(PIA)for zero-sum stochastic differential games with the basic long-run average criterion,as well as with its more selective version,the so-called bias criterion.The system is assumed to be a nondegenerate diffusion.We use Lyapunov-like stability conditions that ensure the existence and boundedness of the solution to certain Poisson equation.We also ensure the convergence of a sequence of such solutions,of the corresponding sequence of policies,and,ultimately,of the PIA.

Active Power Correction Strategies Based on Deep Reinforcement Learning Part II:A Distributed Solution for Adaptability

This article is the second part of Active Power Correction Strategies Based on Deep Reinforcement Learning.In Part II,we consider the renewable energy scenarios plugged into the large-scale power grid and provide an adaptive algorithmic implementation to maintain power grid stability.Based on the robustness method in Part I,a distributed deep reinforcement learning method is proposed to overcome the infuence of the increasing renewable energy penetration.A multi-agent system is implemented in multiple control areas of the power system,which conducts a fully cooperative stochastic game.Based on the Monte Carlo tree search mentioned in Part I,we select practical actions in each sub-control area to search the Nash equilibrium of the game.Based on the QMIX method,a structure of offine centralized training and online distributed execution is proposed to employ better practical actions in the active power correction control.Our proposed method is evaluated in the modified global competition scenario cases of“2020 Learning to Run a Power Network.Neurips Track 2”.

Mean-field type FBSDEs in a domination-monotonicity framework and LQ multi-level Stackelberg games

Motivated by various mean-field type linear-quadratic(MF-LQ,for short)multilevel Stackelberg games,we propose a kind of multi-level self-similar randomized dominationmonotonicity structures.When the coefficients of a class of mean-field type forwardbackward stochastic differential equations(MF-FBSDEs,for short)satisfy this kind of structures,we prove the existence,the uniqueness,an estimate and the continuous dependence on the coefficients of solutions.Further,the theoretical results are applied to construct unique Stackelberg equilibria for forward and backward MF-LQ multi-level Stackelberg games,respectively.