Deep reinforcement learning using least-squares truncated temporal-difference

Policy evaluation(PE)is a critical sub-problem in reinforcement learning,which estimates the value function for a given policy and can be used for policy improvement.However,there still exist some limitations in current PE methods,such as low sample efficiency and local convergence,especially on complex tasks.In this study,a novel PE algorithm called Least-Squares Truncated Temporal-Difference learning(LST2D)is proposed.In LST2D,an adaptive truncation mechanism is designed,which effectively takes advantage of the fast convergence property of Least-Squares Temporal Difference learning and the asymptotic convergence property of Temporal Difference learning(TD).Then,two feature pre-training methods are utilised to improve the approximation ability of LST2D.Furthermore,an Actor-Critic algorithm based on LST2D and pre-trained feature representations(ACLPF)is proposed,where LST2D is integrated into the critic network to improve learning-prediction efficiency.Comprehensive simulation studies were conducted on four robotic tasks,and the corresponding results illustrate the effectiveness of LST2D.The proposed ACLPF algorithm outperformed DQN,ACER and PPO in terms of sample efficiency and stability,which demonstrated that LST2D can be applied to online learning control problems by incorporating it into the actor-critic architecture.

Sub-Differential Characterizations of Non-Smooth Lower Semi-Continuous Pseudo-Convex Functions on Real Banach Spaces

In this paper, we characterize lower semi-continuous pseudo-convex functions f : X → R ∪ {+ ∞} on convex subset of real Banach spaces K ⊂ X with respect to the pseudo-monotonicity of its Clarke-Rockafellar Sub-differential. We extend the results on the characterizations of non-smooth convex functions f : X → R ∪ {+ ∞} on convex subset of real Banach spaces K ⊂ X with respect to the monotonicity of its sub-differentials to the lower semi-continuous pseudo-convex functions on real Banach spaces.

Stochastic level-value approximation for quadratic integer convex programming

We propose a stochastic level value approximation method for a quadratic integer convex minimizing problem in this paper. This method applies an importance sampling technique, and make use of the cross-entropy method to update the sample density functions. We also prove the asymptotic convergence of this algorithm, and report some numerical results to illuminate its effectiveness.

ON VECTOR VALUED FUNCTION APPROXIMATION USING A PEAK NORM

A peak norm is defined for Lp spaces of E-valued Bochner integrable functions, where E is a Banach space, and best approximations from a sun to elements of the space are characterized. Applications are given to some families of simultaneous best approximation problems.

Hierarchical fuzzy ART for Q-learning and its application in air combat simulation

Value function approximation plays an important role in reinforcement learning(RL)with continuous state space,which is widely used to build decision models in practice.Many traditional approaches require experienced designers to manually specify the formulization of the approximating function,leading to the rigid,non-adaptive representation of the value function.To address this problem,a novel Q-value function approximation method named‘Hierarchical fuzzy Adaptive Resonance Theory’(HiART)is proposed in this paper.HiART is based on the Fuzzy ART method and is an adaptive classification network that learns to segment the state space by classifying the training input automatically.HiART begins with a highly generalized structure where the number of the category nodes is limited,which is beneficial to speed up the learning process at the early stage.Then,the network is refined gradually by creating the attached subnetworks,and a layered network structure is formed during this process.Based on this adaptive structure,HiART alleviates the dependence on expert experience to design the network parameter.The effectiveness and adaptivity of HiART are demonstrated in the Mountain Car benchmark problem with both fast learning speed and low computation time.Finally,a simulation application example of the one versus one air combat decision problem illustrates the applicability of HiART.

DISCRETE APPROXIMATIONS FOR SINGULARLY PERTURBED BOUNDARY VALUE PROBLEMS WITH PARABOLIC LAYERS,Ⅱ

In his series of three papers we study singularly perturbed (SP) boundary valueproblems for equations of elliptic and parabolic type. For small values of the pertur-bation parameter parabolic boundary and interior layers appear in these problems.If classical discretisation methods are used, the solution of the finite differencescheme and the approximation of the diffusive flux do not converge uniformly withrespect to this parameter. Using the method of special, adapted grids, we canconstruct difference schemes that allow approximation of the solution and the nor-malised diffusive flux uniformly with respect to the small parameter.We also consider sillgularly perturbed boundary value problems for convection-diffusion equations. Also for these problems we construct special finite differenceschemes, the solution of which converges ε-uniformly We study what problems ap-pear, when classical schemes are used for the approximation of the spatial deriva-tives. We compare the results with those obtained by the adapted approach. Re-sults of numerical experiments are discussed.In the three papers we first give an introduction on the general problem, andthen we consider respectively (i) Problems for SP parabolic equations, for whichthe solution and the normalised diffusive fluxes are required; (ii) Problems for SPelliptic equations with boundary conditions of Diriclilet, Neumann and RDbin type;(iii) Problems for SP parabolic equation with discontinuous boundary conditions-

Interprocedural Constant Range Propagation and Alias Analysis by Multiple Version Method

A set of methods for interprocedural analysis is proposed. First, an ap-proach for interprocedural constant propagation is given. Then the concept of constant propagation is extended so as to meet the needs of data dependence analysis. Besides certain constant, constant range can also be propagated. The related propagating rules are introduced, and an idea for computing Return function is given. This approach can solve almost all interprocedural constant propagation problems with non-recursive calls. Second, a muItiple-version par-allelizing technique is also proposed for alias problem. The work related to this paper has been implemented on a shared-memory parallel computer.

DISCRETE APPROXIMATIONS FOR SINGULARLY PERTURBED BOUNDARY VALUE PEOBLEMS WITH PARABOLIC LAYERS,I

In this series of three papers we study singularly perturbed (SP) boundaryvalue problems for equations of eiliptic and parabolic type- For small values ofthe perturbation parameter parabolic boundary and interior layers appear in theseproblems. If classical discretisation methods are used, the solution of the finitedifference scheme and the approximation of the diffusive flux do not converge uniformly with respect to this parameter. Using the method of special, edapted grids,we can construct difference schemes that allow apprcximation of the solution andthe normalised diffusive flux uniformly with respect to the small parameter.We also consider singularly perturbed boundary value problems for convection-diffusion equations. Also for these problems we construct special finite differenceschemes, the solution of which converges ε-uniformly. We study what problems appear, when classical schemes are used for the approximation of the spatial derivatives. We compare the results with those obtained by the adapted approach. Re-sults of numerical experiments are discussed.In the three papers we first give an introduction on the general problem, andthen we consider respectively (i) Problems for SP parabolic equations, for whichthe solution and the normalised diffusive fluxes are required; (ii) Problems for SPelliptic equations with boundary conditions of Dirichlet, Neumann and Robin type;(iii) Problems for SP parabolic equation with discontinuous boundary conditions.

DISCRETE APPROXIMATIONS FOR SINGULARLY PERTURBED BOUNDARY VALUE PROBLEMS WITH PARABOLIC LAYERS, III

In this series of three papers we study singularly perturbed (SP) boundary vaue problems for equations of elliptic and parabolic troe. For small values of the perturbation parameter parabolic boundary and interior layers appear in these problems. If classical discretisation methods are used, the solution of the finite difference scheme and the approximation of the diffusive flux do not converge uniformly with respect to this parameter. Using the method of special, adapted grids,we can construct difference schemes that allow approkimation of the solution and the normalised diffusive flux uniformly with respect to the small parameter.We also consider singularly perturbed boundary value problems for convection diffusion equations. Also for these problems we construct special finite difference schemes, the solution of which converges E-uniformly We study what problems appear, when classical schemes are used for the approximation of the spatial deriva tives. We compare the results with those obtained by the adapted approach. Results of numerical experiments are discussed.In the three papers we first give an introduction on the general problem, and then we consider respectively (i) Problems for SP parabolic equations, for which the solution and the normalised diffusive fluxes are required; (ii) Problems for SP elliptic equations with boundary conditions of Dirichlet, Neumann and Robin type;(iii) Problems for SP parabolic eqllation with discontinuous boundaxy conditions