Q-greedyUCB: a New Exploration Policy to Learn Resource-Efficient Scheduling

This paper proposes a Reinforcement learning(RL)algorithm to find an optimal scheduling policy to minimize the delay for a given energy constraint in communication system where the environments such as traffic arrival rates are not known in advance and can change over time.For this purpose,this problem is formulated as an infinite-horizon Constrained Markov Decision Process(CMDP).To handle the constrained optimization problem,we first adopt the Lagrangian relaxation technique to solve it.Then,we propose a variant of Q-learning,Q-greedyUCB that combinesε-greedy and Upper Confidence Bound(UCB)algorithms to solve this constrained MDP problem.We mathematically prove that the Q-greedyUCB algorithm converges to an optimal solution.Simulation results also show that Q-greedyUCB finds an optimal scheduling strategy,and is more efficient than Q-learning withε-greedy,R-learning and the Averagepayoff RL(ARL)algorithm in terms of the cumulative regret.We also show that our algorithm can learn and adapt to the changes of the environment,so as to obtain an optimal scheduling strategy under a given power constraint for the new environment.

Mixed deterministic and random optimal control of linear stochastic systems with quadratic costs

In this paper,we consider the mixed optimal control of a linear stochastic system with a quadratic cost functional,with two controllers—one can choose only deterministic time functions,called the deterministic controller,while the other can choose adapted random processes,called the random controller.The optimal control is shown to exist under suitable assumptions.The optimal control is characterized via a system of fully coupled forward-backward stochastic differential equations(FBSDEs)of mean-field type.We solve the FBSDEs via solutions of two(but decoupled)Riccati equations,and give the respective optimal feedback law for both deterministic and random controllers,using solutions of both Riccati equations.The optimal state satisfies a linear stochastic differential equation(SDE)of mean-field type.Both the singular and infinite time-horizonal cases are also addressed.

LINEAR QUADRATIC REGULATION FOR DISCRETE-TIME SYSTEMS WITH INPUT DELAY:SPECTRAL FACTORIZATION APPROACH

The infinite-horizon linear quadratic regulation （LQR） problem is settled for discretetime systems with input delay. With the help of an autoregressive moving average （ARMA） innovation model, solutions to the underlying problem are obtained. The design of the optimal control law involves in resolving one polynomial equation and one spectral factorization. The latter is the major obstacle of the present problem, and the reorganized innovation approach is used to clear it up. The calculation of spectral factorization finally comes down to solving two Riccati equations with the same dimension as the original systems.