Optimizing the Multi-Objective Discrete Particle Swarm Optimization Algorithm by Deep Deterministic Policy Gradient Algorithm

Deep deterministic policy gradient(DDPG)has been proved to be effective in optimizing particle swarm optimization(PSO),but whether DDPG can optimize multi-objective discrete particle swarm optimization(MODPSO)remains to be determined.The present work aims to probe into this topic.Experiments showed that the DDPG can not only quickly improve the convergence speed of MODPSO,but also overcome the problem of local optimal solution that MODPSO may suffer.The research findings are of great significance for the theoretical research and application of MODPSO.

Perception Enhanced Deep Deterministic Policy Gradient for Autonomous Driving in Complex Scenarios

Autonomous driving has witnessed rapid advancement;however,ensuring safe and efficient driving in intricate scenarios remains a critical challenge.In particular,traffic roundabouts bring a set of challenges to autonomous driving due to the unpredictable entry and exit of vehicles,susceptibility to traffic flow bottlenecks,and imperfect data in perceiving environmental information,rendering them a vital issue in the practical application of autonomous driving.To address the traffic challenges,this work focused on complex roundabouts with multi-lane and proposed a Perception EnhancedDeepDeterministic Policy Gradient(PE-DDPG)for AutonomousDriving in the Roundabouts.Specifically,themodel incorporates an enhanced variational autoencoder featuring an integrated spatial attention mechanism alongside the Deep Deterministic Policy Gradient framework,enhancing the vehicle’s capability to comprehend complex roundabout environments and make decisions.Furthermore,the PE-DDPG model combines a dynamic path optimization strategy for roundabout scenarios,effectively mitigating traffic bottlenecks and augmenting throughput efficiency.Extensive experiments were conducted with the collaborative simulation platform of CARLA and SUMO,and the experimental results show that the proposed PE-DDPG outperforms the baseline methods in terms of the convergence capacity of the training process,the smoothness of driving and the traffic efficiency with diverse traffic flow patterns and penetration rates of autonomous vehicles(AVs).Generally,the proposed PE-DDPGmodel could be employed for autonomous driving in complex scenarios with imperfect data.

RIS-Assisted UAV-D2D Communications Exploiting Deep Reinforcement Learning

Device-to-device(D2D)communications underlying cellular networks enabled by unmanned aerial vehicles(UAV)have been regarded as promising techniques for next-generation communications.To mitigate the strong interference caused by the line-of-sight(LoS)airto-ground channels,we deploy a reconfigurable intelligent surface(RIS)to rebuild the wireless channels.A joint optimization problem of the transmit power of UAV,the transmit power of D2D users and the RIS phase configuration are investigated to maximize the achievable rate of D2D users while satisfying the quality of service(QoS)requirement of cellular users.Due to the high channel dynamics and the coupling among cellular users,the RIS,and the D2D users,it is challenging to find a proper solution.Thus,a RIS softmax deep double deterministic(RIS-SD3)policy gradient method is proposed,which can smooth the optimization space as well as reduce the number of local optimizations.Specifically,the SD3 algorithm maximizes the reward of the agent by training the agent to maximize the value function after the softmax operator is introduced.Simulation results show that the proposed RIS-SD3 algorithm can significantly improve the rate of the D2D users while controlling the interference to the cellular user.Moreover,the proposed RIS-SD3 algorithm has better robustness than the twin delayed deep deterministic(TD3)policy gradient algorithm in a dynamic environment.

Deep reinforcement learning and its application in autonomous fitting optimization for attack areas of UCAVs

The ever-changing battlefield environment requires the use of robust and adaptive technologies integrated into a reliable platform. Unmanned combat aerial vehicles(UCAVs) aim to integrate such advanced technologies while increasing the tactical capabilities of combat aircraft. As a research object, common UCAV uses the neural network fitting strategy to obtain values of attack areas. However, this simple strategy cannot cope with complex environmental changes and autonomously optimize decision-making problems. To solve the problem, this paper proposes a new deep deterministic policy gradient(DDPG) strategy based on deep reinforcement learning for the attack area fitting of UCAVs in the future battlefield. Simulation results show that the autonomy and environmental adaptability of UCAVs in the future battlefield will be improved based on the new DDPG algorithm and the training process converges quickly. We can obtain the optimal values of attack areas in real time during the whole flight with the well-trained deep network.

Distributed optimization of electricity-Gas-Heat integrated energy system with multi-agent deep reinforcement learning

The coordinated optimization problem of the electricity-gas-heat integrated energy system(IES)has the characteristics of strong coupling,non-convexity,and nonlinearity.The centralized optimization method has a high cost of communication and complex modeling.Meanwhile,the traditional numerical iterative solution cannot deal with uncertainty and solution efficiency,which is difficult to apply online.For the coordinated optimization problem of the electricity-gas-heat IES in this study,we constructed a model for the distributed IES with a dynamic distribution factor and transformed the centralized optimization problem into a distributed optimization problem in the multi-agent reinforcement learning environment using multi-agent deep deterministic policy gradient.Introducing the dynamic distribution factor allows the system to consider the impact of changes in real-time supply and demand on system optimization,dynamically coordinating different energy sources for complementary utilization and effectively improving the system economy.Compared with centralized optimization,the distributed model with multiple decision centers can achieve similar results while easing the pressure on system communication.The proposed method considers the dual uncertainty of renewable energy and load in the training.Compared with the traditional iterative solution method,it can better cope with uncertainty and realize real-time decision making of the system,which is conducive to the online application.Finally,we verify the effectiveness of the proposed method using an example of an IES coupled with three energy hub agents.

Moving target defense of routing randomization with deep reinforcement learning against eavesdropping attack

Eavesdropping attacks have become one of the most common attacks on networks because of their easy implementation. Eavesdropping attacks not only lead to transmission data leakage but also develop into other more harmful attacks. Routing randomization is a relevant research direction for moving target defense, which has been proven to be an effective method to resist eavesdropping attacks. To counter eavesdropping attacks, in this study, we analyzed the existing routing randomization methods and found that their security and usability need to be further improved. According to the characteristics of eavesdropping attacks, which are “latent and transferable”, a routing randomization defense method based on deep reinforcement learning is proposed. The proposed method realizes routing randomization on packet-level granularity using programmable switches. To improve the security and quality of service of legitimate services in networks, we use the deep deterministic policy gradient to generate random routing schemes with support from powerful network state awareness. In-band network telemetry provides real-time, accurate, and comprehensive network state awareness for the proposed method. Various experiments show that compared with other typical routing randomization defense methods, the proposed method has obvious advantages in security and usability against eavesdropping attacks.

A UAV collaborative defense scheme driven by DDPG algorithm

The deep deterministic policy gradient(DDPG)algo-rithm is an off-policy method that combines two mainstream reinforcement learning methods based on value iteration and policy iteration.Using the DDPG algorithm,agents can explore and summarize the environment to achieve autonomous deci-sions in the continuous state space and action space.In this paper,a cooperative defense with DDPG via swarms of unmanned aerial vehicle(UAV)is developed and validated,which has shown promising practical value in the effect of defending.We solve the sparse rewards problem of reinforcement learning pair in a long-term task by building the reward function of UAV swarms and optimizing the learning process of artificial neural network based on the DDPG algorithm to reduce the vibration in the learning process.The experimental results show that the DDPG algorithm can guide the UAVs swarm to perform the defense task efficiently,meeting the requirements of a UAV swarm for non-centralization,autonomy,and promoting the intelligent development of UAVs swarm as well as the decision-making process.

Low Carbon Economic Dispatch of Integrated Energy System Considering Power Supply Reliability and Integrated Demand Response

Integrated energy system optimization scheduling can improve energy efficiency and low carbon economy.This paper studies an electric-gas-heat integrated energy system,including the carbon capture system,energy coupling equipment,and renewable energy.An energy scheduling strategy based on deep reinforcement learning is proposed to minimize operation cost,carbon emission and enhance the power supply reliability.Firstly,the lowcarbon mathematical model of combined thermal and power unit,carbon capture system and power to gas unit(CCP)is established.Subsequently,we establish a low carbon multi-objective optimization model considering system operation cost,carbon emissions cost,integrated demand response,wind and photovoltaic curtailment,and load shedding costs.Furthermore,considering the intermittency of wind power generation and the flexibility of load demand,the low carbon economic dispatch problem is modeled as a Markov decision process.The twin delayed deep deterministic policy gradient(TD3)algorithm is used to solve the complex scheduling problem.The effectiveness of the proposed method is verified in the simulation case studies.Compared with TD3,SAC,A3C,DDPG and DQN algorithms,the operating cost is reduced by 8.6%,4.3%,6.1%and 8.0%.

Deep Reinforcement Learning Enabled Bi-level Robust Parameter Optimization of Hydropower-dominated Systems for Damping Ultra-low Frequency Oscillation

This paper proposes a robust and computationally efficient control method for damping ultra-low frequency oscillations(ULFOs) in hydropower-dominated systems. Unlike the existing robust optimization based control formulation that can only deal with a limited number of operating conditions, the proposed method reformulates the control problem into a bi-level robust parameter optimization model. This allows us to consider a wide range of system operating conditions. To speed up the bi-level optimization process, the deep deterministic policy gradient(DDPG) based deep reinforcement learning algorithm is developed to train an intelligent agent. This agent can provide very fast lower-level decision variables for the upper-level model, significantly enhancing its computational efficiency. Simulation results demonstrate that the proposed method can achieve much better damping control performance than other alternatives with slightly degraded dynamic response performance of the governor under various types of operating conditions.

Optimal control of a hybrid microgrid for hydrogenbased heat supply using deep reinforcement learning

Green hydrogen is considered one of the key technologies of the energy transition,as it can be used to store surpluses from renewable energies in times of high solar radiation or wind speed for use in dark lulls.This paper examines the decarbonization potential of hydrogen for the heating industry.Worldwide,99%of hydrogen is produced from fossil fuels,because hydrogen derived from renew-able energy sources remains prohibitively expensive compared with its conventional counterpart.However,due to the expansion of renewable energy sources and the current energy crisis of conventional energy sources,hydrogen from renewable energy sources is becoming more and more economical.To optimize the efficiency of green hydrogen production and make it more price-competitive,the author simulates a hydrogen production plant consisting of a photovoltaic plant,a power grid,hydrogen storage,an electro-lyser,a natural gas purchase option,a district heating plant and households.Using the deep deterministic policy gradient algorithm from deep reinforcement learning,the plant is designed to optimize itself by simulating different production scenarios and deriving strategies.The connected district heating plant is used to map how hydrogen can be optimally used for heat supply.A demonstrable outcome of this paper is that the utilization of deep deterministic policy gradient,over the course of a full year,can result in a com-petitive production of hydrogen derived from renewable or stored energy sources for the heating industry as a natural gas substitute.

On‑Ramp Merging for Highway Autonomous Driving:An Application of a New Safety Indicator in Deep Reinforcement Learning

This paper proposes an improved decision-making method based on deep reinforcement learning to address on-ramp merging challenges in highway autonomous driving.A novel safety indicator,time difference to merging(TDTM),is introduced,which is used in conjunction with the classic time to collision(TTC)indicator to evaluate driving safety and assist the merging vehicle in finding a suitable gap in traffic,thereby enhancing driving safety.The training of an autonomous driving agent is performed using the Deep Deterministic Policy Gradient(DDPG)algorithm.An action-masking mechanism is deployed to prevent unsafe actions during the policy exploration phase.The proposed DDPG+TDTM+TTC solution is tested in on-ramp merging scenarios with different driving speeds in SUMO and achieves a success rate of 99.96%without significantly impacting traffic efficiency on the main road.The results demonstrate that DDPG+TDTM+TTC achieved a higher on-ramp merging success rate of 99.96%compared to DDPG+TTC and DDPG.

Relevant experience learning:A deep reinforcement learning method for UAV autonomous motion planning in complex unknown environments

Unmanned Aerial Vehicles(UAVs)play a vital role in military warfare.In a variety of battlefield mission scenarios,UAVs are required to safely fly to designated locations without human intervention.Therefore,finding a suitable method to solve the UAV Autonomous Motion Planning(AMP)problem can improve the success rate of UAV missions to a certain extent.In recent years,many studies have used Deep Reinforcement Learning(DRL)methods to address the AMP problem and have achieved good results.From the perspective of sampling,this paper designs a sampling method with double-screening,combines it with the Deep Deterministic Policy Gradient(DDPG)algorithm,and proposes the Relevant Experience Learning-DDPG(REL-DDPG)algorithm.The REL-DDPG algorithm uses a Prioritized Experience Replay(PER)mechanism to break the correlation of continuous experiences in the experience pool,finds the experiences most similar to the current state to learn according to the theory in human education,and expands the influence of the learning process on action selection at the current state.All experiments are applied in a complex unknown simulation environment constructed based on the parameters of a real UAV.The training experiments show that REL-DDPG improves the convergence speed and the convergence result compared to the state-of-the-art DDPG algorithm,while the testing experiments show the applicability of the algorithm and investigate the performance under different parameter conditions.

Path Following Control for UAV Using Deep Reinforcement Learning Approach

Unmanned aerial vehicles(UAVs)have been extensively used in civil and industrial applications due to the rapid development of the guidance,navigation and control(GNC)technologies.Especially,using deep reinforcement learning methods for motion control acquires a major progress recently,since deep Q-learning algorithm has been successfully applied to the continuous action domain problem.This paper proposes an improved deep deterministic policy gradient(DDPG)algorithm for path following control problem of UAV.A speci-c reward function is designed for minimizing the cross-track error of the path following problem.In the training phase,a double experience replay bu®er(DERB)is used to increase the learning e±ciency and accelerate the convergence speed.First,the model of UAV path following problem has been established.After that,the framework of DDPG algorithm is constructed.Then the state space,action space and reward function of the UAV path following algorithm are designed.DERB is proposed to accelerate the training phase.Finally,simulation results are carried out to show the e®ectiveness of the proposed DERB–DDPG method.

Deep reinforcement learning for dynamic computation offloading and resource allocation in cache-assisted mobile edge computing systems

Mobile Edge Computing(MEC)is one of the most promising techniques for next-generation wireless communication systems.In this paper,we study the problem of dynamic caching,computation offloading,and resource allocation in cache-assisted multi-user MEC systems with stochastic task arrivals.There are multiple computationally intensive tasks in the system,and each Mobile User(MU)needs to execute a task either locally or remotely in one or more MEC servers by offloading the task data.Popular tasks can be cached in MEC servers to avoid duplicates in offloading.The cached contents can be either obtained through user offloading,fetched from a remote cloud,or fetched from another MEC server.The objective is to minimize the long-term average of a cost function,which is defined as a weighted sum of energy consumption,delay,and cache contents’fetching costs.The weighting coefficients associated with the different metrics in the objective function can be adjusted to balance the tradeoff among them.The optimum design is performed with respect to four decision parameters:whether to cache a given task,whether to offload a given uncached task,how much transmission power should be used during offloading,and how much MEC resources to be allocated for executing a task.We propose to solve the problems by developing a dynamic scheduling policy based on Deep Reinforcement Learning(DRL)with the Deep Deterministic Policy Gradient(DDPG)method.A new decentralized DDPG algorithm is developed to obtain the optimum designs for multi-cell MEC systems by leveraging on the cooperations among neighboring MEC servers.Simulation results demonstrate that the proposed algorithm outperforms other existing strategies,such as Deep Q-Network(DQN).

A novel movies recommendation algorithm based on reinforcement learning with DDPG policy

Purpose-English original movies played an important role in English learning and communication.In order to find the required movies for us from a large number of English original movies and reviews,this paper proposed an improved deep reinforcement learning algorithm for the recommendation of movies.In fact,although the conventional movies recommendation algorithms have solved the problem of information overload,they still have their limitations in the case of cold start-up and sparse data.Design/methodology/approach-To solve the aforementioned problems of conventional movies recommendation algorithms,this paper proposed a recommendation algorithm based on the theory of deep reinforcement learning,which uses the deep deterministic policy gradient(DDPG)algorithm to solve the cold starting and sparse data problems and uses Item2vec to transform discrete action space into a continuous one.Meanwhile,a reward function combining with cosine distance and Euclidean distance is proposed to ensure that the neural network does not converge to local optimum prematurely.Findings-In order to verify the feasibility and validity of the proposed algorithm,the state of the art and the proposed algorithm are compared in indexes of RMSE,recall rate and accuracy based on the MovieLens English original movie data set for the experiments.Experimental results have shown that the proposed algorithm is superior to the conventional algorithm in various indicators.Originality/value-Applying the proposed algorithm to recommend English original movies,DDPG policy produces better recommendation results and alleviates the impact of cold start and sparse data.

Reinforcement learning-based missile terminal guidance of maneuvering targets with decoys

In this paper,a missile terminal guidance law based on a new Deep Deterministic Policy Gradient(DDPG)algorithm is proposed to intercept a maneuvering target equipped with an infrared decoy.First,to deal with the issue that the missile cannot accurately distinguish the target from the decoy,the energy center method is employed to obtain the equivalent energy center(called virtual target)of the target and decoy,and the model for the missile and the virtual decoy is established.Then,an improved DDPG algorithm is proposed based on a trusted-search strategy,which significantly increases the train efficiency of the previous DDPG algorithm.Furthermore,combining the established model,the network obtained by the improved DDPG algorithm and the reward function,an intelligent missile terminal guidance scheme is proposed.Specifically,a heuristic reward function is designed for training and learning in combat scenarios.Finally,the effectiveness and robustness of the proposed guidance law are verified by Monte Carlo tests,and the simulation results obtained by the proposed scheme and other methods are compared to further demonstrate its superior performance.

A DDPG-based solution for optimal consensus of continuous-time linear multi-agent systems

Modeling a system in engineering applications is a time-consuming and labor-intensive task,as system parameters may change with temperature,component aging,etc.In this paper,a novel data-driven model-free optimal controller based on deep deterministic policy gradient(DDPG)is proposed to address the problem of continuous-time leader-following multi-agent consensus.To deal with the problem of the dimensional explosion of state space and action space,two different types of neural nets are utilized to fit them instead of the time-consuming state iteration process.With minimal energy consumption,the proposed controller achieves consensus only based on the consensus error and does not require any initial admissible policies.Besides,the controller is self-learning,which means it can achieve optimal control by learning in real time as the system parameters change.Finally,the proofs of convergence and stability,as well as some simulation experiments,are provided to verify the algorithm’s effectiveness.

Jointly improving energy efficiency and smoothing power oscillations of integrated offshore wind and photovoltaic power: a deep reinforcement learning approach

This paper proposes a novel deep reinforcement learning(DRL)control strategy for an integrated offshore wind and photovoltaic(PV)power system for improving power generation efficiency while simultaneously damping oscilla-tions.A variable-speed offshore wind turbine(OWT)with electrical torque control is used in the integrated offshore power system whose dynamic models are detailed.By considering the control system as a partially-observable Markov decision process,an actor-critic architecture model-free DRL algorithm,namely,deep deterministic policy gradient,is adopted and implemented to explore and learn the optimal multi-objective control policy.The potential and effectiveness of the integrated power system are evaluated.The results imply that an OWT can respond quickly to sudden changes of the inflow wind conditions to maximize total power generation.Significant oscillations in the overall power output can also be well suppressed by regulating the generator torque,which further indicates that complementary operation of offshore wind and PV power can be achieved.

Deep reinforcement learning for online scheduling of photovoltaic systems with battery energy storage systems

A new online scheduling algorithm is proposed for photovoltaic(PV)systems with battery-assisted energy storage systems(BESS).The stochastic nature of renewable energy sources necessitates the employment of BESS to balance energy supplies and demands under uncertain weather conditions.The proposed online scheduling algorithm aims at minimizing the overall energy cost by performing actions such as load shifting and peak shaving through carefully scheduled BESS charging/discharging activities.The scheduling algorithm is developed by using deep deterministic policy gradient(DDPG),a deep reinforcement learning(DRL)algorithm that can deal with continuous state and action spaces.One of the main contributions of this work is a new DDPG reward function,which is designed based on the unique behaviors of energy systems.The new reward function can guide the scheduler to learn the appropriate behaviors of load shifting and peak shaving through a balanced process of exploration and exploitation.The new scheduling algorithm is tested through case studies using real world data,and the results indicate that it outperforms existing algorithms such as Deep Q-learning.The online algorithm can efficiently learn the behaviors of optimum non-casual off-line algorithms.

Data-driven and Model-based Hybrid Reinforcement Learning to Reduce Stress on Power Systems Branches

This work proposes a reinforcement learning(RL)approach to tackle the control problem of branch overload relief in large power systems.Accordingly,a control agent is trained to change generators'real power output in order to relieve the stressed branches.For large power systems,this control problem becomes one whose decision space(i.e.,the action space)is both highly-dimensioned and continuous.This makes it extremely difficult to have successful training for RL-based agents.To improve the effectiveness,a data-driven and model-based hybrid approach is proposed to optimize the control by combining RL-agent actions and generator shifting factor-driven actions.Accordingly,with the proposed approach the RL-agent successfully trains on large power systems.The proposed design is tested on both the IEEE 118-bus testing system and a 2749-bus real system.The obtained results show that the proposed hybrid approach outperforms the data-driven training approach.