Sampled-data control through model-free reinforcement learning with effective experience replay

Reinforcement Learning(RL)based control algorithms can learn the control strategies for nonlinear and uncertain environment during interacting with it.Guided by the rewards generated by environment,a RL agent can learn the control strategy directly in a model-free way instead of investigating the dynamic model of the environment.In the paper,we propose the sampled-data RL control strategy to reduce the computational demand.In the sampled-data control strategy,the whole control system is of a hybrid structure,in which the plant is of continuous structure while the controller(RL agent)adopts a discrete structure.Given that the continuous states of the plant will be the input of the agent,the state–action value function is approximated by the fully connected feed-forward neural networks(FCFFNN).Instead of learning the controller at every step during the interaction with the environment,the learning and acting stages are decoupled to learn the control strategy more effectively through experience replay.In the acting stage,the most effective experience obtained during the interaction with the environment will be stored and during the learning stage,the stored experience will be replayed to customized times,which helps enhance the experience replay process.The effectiveness of proposed approach will be verified by simulation examples.

Characteristics of gravity signal and loading effect in China

The complex geographical environment in China makes its gravity signals miscellaneous.This work gives a comprehensive representation and explanation in secular trend of gravity change in different regions,the key features of which include positive trend in inner Tibet Plateau and South China and negative trend in North China plain and high mountain Asia（HMA）.We also present the patterns of amplitudes and phases of annual and semiannual change.The mechanism underlying the semiannual period is explicitly discussed.The displacement in three directions expressed in terms of geo-potential spherical coefficients and load Love numbers are given.A case study applied with these equations is presented.The results show that Global Positioning System（GPS） observations can be used to compare with Gravity Recovery and Climate Experiment（GRACE） derived displacement and the vertical direction has a signal-noise-ratio of about one order of magnitude larger than the horizontal directions.

A Fluid Experiment of Large-Scale Topography Effect on Baroclinic Wave Flows

The effects of topography on baroclinic wave flows are studied experimentally in a thermally driven rotating annulus of fluid.Fourier analysis and complex principal component (CPC) analysis of the experimental data show that, due to topographic forcing, the flow is bimodal rather than a single mode. Under suitable imposed experimental parameters, near thermal Rossby number ROT = 0.1 and Taylor number Ta = 2.2 × 107, the large-scale topography produces low-frequency oscillation in the flow and rather long-lived flow pattern resembling blocking in the atmospheric circulation. The 'blocking' phenomenon is caused by the resonance of travelling waves and the quasi-stationary waves forced by topography.The large-scale topography transforms wavenumber-homogeneous flows into wavenumber-dispersed flows, and the dispersed flows possess lower wavenumbers.

Some Effects of Rotation Rate on Planetary-Scale Wave Flows

A series of experiments were performed in a rotating annulus of fluid to study effects of rotation rate on pianeta ry-scale baroclinic wave flows. The experiments reveal that change in rotation rate of fluid container causes variation in Rossby number and Taylor number in flows and leads to change in flow patterns and in phase and amplitude of quasi-stationary waves. For instance, with increasing rotation rate, amplitude of quasi-stationary waves increases and phase shifts upstream. On the contrary, with decreasing rotation rate, amplitude of quasi-stationary waves de creases and phase shifts downstream. In the case of the earth's atmosphere, although magnitude of variation in earth's rotation rate is very small, yet it causes a very big change in zonal velocity component of wind in the atmosphere and of currents in the ocean, and therefore causes a remarkable change in Rossby number and Taylor number determining regimes in planetary-scale geophysical flows. 1 he observation reveals that intensity and geographic location of subtropic anticyclones in both of the Northern and Southern Hemispheres change consistently with the variation in earth's rotation rate. The results of fluid experiments are consistent, qualitatively, with observed phenomena in the atmospheric circulation.

Recent Progress of the HL-2A Multi-Channel HCOOH Laser Interferometer/Polarimeter

A multichannel methanoic acid （HCOOH, λ=432.5 μm） laser interferometer/polarimeter is being developed from the previous eight-channel hydrogen cyanide （HCN, λ=337 μm） laser interferometer in the HL-2A tokamak. A conventional Michelson-type interometer is used for the electron density measurement, and a Dodel-Kunz-type polarimeter is used for the Faraday rotation effect measurement, respectively. Each HCOOH laser can produce a linearly polarized radiation at a power lever of -30 mW, and a power stability 〈10% in 50 rain. A beam waist （diameter d0 ≈12.0 mm, about 200 mm away from the outlet） is finally determined through a chopping modulation technique. The latest optical layout of the interferometer/polarimeter has been finished, and the hardware data processing system based on the fast Fourier transform phase- comparator technique is being explored. In order to demonstrate the feasibility of the diagnostic scheme, two associated bench simulation experiments were carried out in the laboratory, in which the plasma was simulated by a piece of polytetrafluoroethene plate, and the Faraday rotation effect was simulated by a rotating half-wave plate. Simulation results agreed well with the initial experimental conditions. At present, the HCOOH laser interferometer/polarimeter system is being assembled on HL-2A, and is planned to be applied in the 2014-2015 experimental campaign.

NUMERICAL EXPERIMENT WITH PROCESSES FOR EFFECT OF AUSTRALIAN COLD AIR ACTIVITY ON EAST-ASIAN SUMMER MONSOON

Based on diagnostic results,a numerical study is made of the processes of Australian cold air activity affecting East Asian summer monsoon by using Kuo-Qian P-σ incorporated coordinate five-layer primitive equation spherical band model.Analysis is done of the response to the Southern Hemisphere circulation with and without cold air activity in Australia of the flow,rainfall and diabatic heating fields in the monsoon area of Asia,especially,East Asia,with special attention to the intensification and northward march of the monsoon due to the activity.It is found that the processes for the effect transmission are very analogous to the meridional propagation of quasi-40-day oscillation,together with the meridional wind disturbance showing south-north travel and the flow/rainfall fields exhibitirg corresponding movement in this direction,only with a 12-day lag.

EXPERIMENT STUDY ON THE EFFECT OF SHENG-MAI INJECTION ON THE CONTRACTIVITY OF DIAPHRAGM AND ITS MECHANISM

In this study the effect of Sheng-ai Injection i.e. Red Ginseng-Ophiopogon Root Injection (one kind of traditional Chinese medicines) on the contractivity of diaphragm was observed. The results confirmed that Sheng-ai Injection increased Pdi of the fatigued diaphragm in rabbits and reduced the time needed for the recovery of Pdi of fatigued diaphragm to the normal value. These results suggest that Sheng-Mai Injection can increase the contractive force and promote the recovery of the fatigued diaphragm. The effect of Sheng-ai Injection on the contractivity of the isolated diaphragmatic bundle of rats was also observed and the results confirmed that Sheng-ai Injection increased the diaphragmatic contractive force directly. This effect of increasing the contractive force of diaphragm was attenuated by adding calcium channel blocker isoptin and disappeared when there was no calcium in the extracellular fluid. It is deduced, therefore, that the mechanism of the effect of Sheng-mai Injection is related to the increased influx of calcium from extracellular fluid into the cells.