Reinforcement Learning-Based Energy Management for Hybrid Power Systems:State-of-the-Art Survey,Review,and Perspectives

The new energy vehicle plays a crucial role in green transportation,and the energy management strategy of hybrid power systems is essential for ensuring energy-efficient driving.This paper presents a state-of-the-art survey and review of reinforcement learning-based energy management strategies for hybrid power systems.Additionally,it envisions the outlook for autonomous intelligent hybrid electric vehicles,with reinforcement learning as the foundational technology.First of all,to provide a macro view of historical development,the brief history of deep learning,reinforcement learning,and deep reinforcement learning is presented in the form of a timeline.Then,the comprehensive survey and review are conducted by collecting papers from mainstream academic databases.Enumerating most of the contributions based on three main directions—algorithm innovation,powertrain innovation,and environment innovation—provides an objective review of the research status.Finally,to advance the application of reinforcement learning in autonomous intelligent hybrid electric vehicles,future research plans positioned as“Alpha HEV”are envisioned,integrating Autopilot and energy-saving control.

Smart Energy Management System Using Machine Learning

Energy management is an inspiring domain in developing of renewable energy sources.However,the growth of decentralized energy production is revealing an increased complexity for power grid managers,inferring more quality and reliability to regulate electricity flows and less imbalance between electricity production and demand.The major objective of an energy management system is to achieve optimum energy procurement and utilization throughout the organization,minimize energy costs without affecting production,and minimize environmental effects.Modern energy management is an essential and complex subject because of the excessive consumption in residential buildings,which necessitates energy optimization and increased user comfort.To address the issue of energy management,many researchers have developed various frameworks;while the objective of each framework was to sustain a balance between user comfort and energy consumption,this problem hasn’t been fully solved because of how difficult it is to solve it.An inclusive and Intelligent Energy Management System(IEMS)aims to provide overall energy efficiency regarding increased power generation,increase flexibility,increase renewable generation systems,improve energy consumption,reduce carbon dioxide emissions,improve stability,and reduce energy costs.Machine Learning(ML)is an emerging approach that may be beneficial to predict energy efficiency in a better way with the assistance of the Internet of Energy(IoE)network.The IoE network is playing a vital role in the energy sector for collecting effective data and usage,resulting in smart resource management.In this research work,an IEMS is proposed for Smart Cities(SC)using the ML technique to better resolve the energy management problem.The proposed system minimized the energy consumption with its intelligent nature and provided better outcomes than the previous approaches in terms of 92.11% accuracy,and 7.89% miss-rate.

A Predictive Energy Management Strategies for Mining Dump Trucks

The plug-in hybrid vehicles(PHEV)technology can effectively address the issues of poor dynamics and higher energy consumption commonly found in traditional mining dump trucks.Meanwhile,plug-in hybrid electric trucks can achieve excellent fuel economy through efficient energy management strategies(EMS).Therefore,a series hybrid system is constructed based on a 100-ton mining dump truck in this paper.And inspired by the dynamic programming(DP)algorithm,a predictive equivalent consumption minimization strategy(P-ECMS)based on the DP optimization result is proposed.Based on the optimal control manifold and the SOC reference trajectory obtained by the DP algorithm,the P-ECMS strategy performs real-time stage parameter optimization to obtain the optimal equivalent factor(EF).Finally,applying the equivalent consumption minimization strategy(ECMS)realizes real-time control.The simulation results show that the equivalent fuel consumption of the P-ECMS strategy under the experimentally collected mining cycle conditions is 150.8 L/100 km,which is 10.9%less than that of the common CDCS strategy(169.3 L/100 km),and achieves 99.47%of the fuel saving effect of the DP strategy(150 L/100 km).

Survey on AI and Machine Learning Techniques for Microgrid Energy Management Systems

In the era of an energy revolution,grid decentralization has emerged as a viable solution to meet the increasing global energy demand by incorporating renewables at the distributed level.Microgrids are considered a driving component for accelerating grid decentralization.To optimally utilize the available resources and address potential challenges,there is a need to have an intelligent and reliable energy management system(EMS)for the microgrid.The artificial intelligence field has the potential to address the problems in EMS and can provide resilient,efficient,reliable,and scalable solutions.This paper presents an overview of existing conventional and AI-based techniques for energy management systems in microgrids.We analyze EMS methods for centralized,decentralized,and distributed microgrids separately.Then,we summarize machine learning techniques such as ANNs,federated learning,LSTMs,RNNs,and reinforcement learning for EMS objectives such as economic dispatch,optimal power flow,and scheduling.With the incorporation of AI,microgrids can achieve greater performance efficiency and more reliability for managing a large number of energy resources.However,challenges such as data privacy,security,scalability,explainability,etc.,need to be addressed.To conclude,the authors state the possible future research directions to explore AI-based EMS's potential in real-world applications.

Distributionally robust optimization based chance-constrained energy management for hybrid energy powered cellular networks

Energy harvesting has been recognized as a promising technique with which to effectively reduce carbon emis-sions and electricity expenses of base stations.However,renewable energy is inherently stochastic and inter-mittent,imposing formidable challenges on reliably satisfying users'time-varying wireless traffic demands.In addition,the probability distribution of the renewable energy or users’wireless traffic demand is not always fully known in practice.In this paper,we minimize the total energy cost of a hybrid-energy-powered cellular network by jointly optimizing the energy sharing among base stations,the battery charging and discharging rates,and the energy purchased from the grid under the constraint of a limited battery size at each base station.In solving the formulated non-convex chance-constrained stochastic optimization problem,a new ambiguity set is built to characterize the uncertainties in the renewable energy and wireless traffic demands according to interval sets of the mean and covariance.Using this ambiguity set,the original optimization problem is transformed into a more tractable second-order cone programming problem by exploiting the distributionally robust optimization approach.Furthermore,a low-complexity distributionally robust chance-constrained energy management algo-rithm,which requires only interval sets of the mean and covariance of stochastic parameters,is proposed.The results of extensive simulation are presented to demonstrate that the proposed algorithm outperforms existing methods in terms of the computational complexity,energy cost,and reliability.

Reinforcement Learning-Based Electric Vehicles Energy Management Strategy with Battery Thermal Model

The promotion of electric vehicles(EVs)is restricted due to their short cruising range.It is desirable to design an effective energy management strategy to improve their energy efficiency.Most existing work concerning energy management strategies focused on hybrids rather than the EVs.The work focusing on the energy management strategy for EVs mainly uses the traditional optimization strategies,thereby limiting the advantages of energy economy.To this end,a novel energy management strategy that considered the impact of battery thermal effects was proposed with the help of reinforcement learning.The main idea was to first analyze the energy flow path of EVs,further formulize the energy management as an optimization problem,and finally propose an online strategy based on reinforcement learning to obtain the optimal strategy.Additionally,extensive simulation results have demonstrated that our strategy reduces energy consumption by at least 27.4%compared to the existing methods.

Energy Management System with Power Offering Strategy for a Microgrid Integrated VPP

In the context of both the Virtual Power Plant (VPP) and microgrid(MG), the Energy Management System (EMS) is a key decision-maker forintegrating Distributed renewable Energy Resources (DERs) efficiently. TheEMS is regarded as a strong enabler of providing the optimized schedulingcontrol in operation and management of usage of disperse DERs and RenewableEnergy reSources (RES) such as a small-size wind-turbine (WT) andphotovoltaic (PV) energies. The main objective to be pursued by the EMSis the minimization of the overall operating cost of the MG integrated VPPnetwork. However, the minimization of the power peaks is a new objective andopen issue to a well-functional EMS, along with the maximization of profitin the energy market. Thus, both objectives have to be taken into accountat the same time. Thus, this paper proposes the EMS application incorporatingpower offering strategy applying a nature-inspired algorithm such asParticle Swarm Optimization (PSO) algorithm, in order to find the optimalsolution of the objective function in the context of the overall operating cost,the coordination of DERs, and the energy losses in a MG integrated VPPnetwork. For a fair DERs coordination with minimized power fluctuationsin the power flow, the power offering strategies with an active power controland re-distribution are proposed. Simulation results show that the proposedMG integrated VPP model with PSO-based EMS employing EgalitarianreDistribution (ED) power offering strategy is most feasible option for theoverall operating cost of VPP revenue. The total operating cost of the proposedEMS with ED strategy is 40.98$ compared to 432.8$ of MGs only withoutEMS. It is concluded that each MGs in the proposed VPP model intelligentlyparticipates in energy trading market compliant with the objective function,to minimize the overall cost and the power fluctuation.

Development of Energy Management System for Micro Grid Operation

The introduction of several small and large-scale industries,malls,shopping complexes,and domestic applications has significantly increased energy consumption.The aim of the work is to simulate a technically viable and economically optimum hybrid power system for residential buildings.The proposed micro-grid model includes four power generators:solar power,wind power,Electricity Board(EB)source,and a Diesel Generator(DG)set,with solar and wind power performing as major sources and the EB supply and DG set serving as backup sources.The core issue in direct current to alternate current conversion is harmonics distortion,a five-stage multilevel inverter is employed with the assistance of an intelligent control system is simulated and the optimum system configuration is estimated to reduce harmonics and improve the power quality.The monthly demand for residential buildings is 13-15 Megawatts.So,almost 433 Kilo-Watts(KW)of electricity is required every day,and if it is used for 8 h per day,50-60 KW of electricity is needed per hour.The overall micro-grid model’s operation and performance are established using MATLAB/SIMULINK software,and simulation results are provided.The simulation results show that the developed system is both cost-effective and environment friendly resulting in yearly cost reductions.

Energy Management and Capacity Optimization of Photovoltaic, Energy Storage System, Flexible Building Power System Considering Combined Benefit

Building structures themselves are one of the key areas of urban energy consumption,therefore,are a major source of greenhouse gas emissions.With this understood,the carbon trading market is gradually expanding to the building sector to control greenhouse gas emissions.Hence,to balance the interests of the environment and the building users,this paper proposes an optimal operation scheme for the photovoltaic,energy storage system,and flexible building power system(PEFB),considering the combined benefit of building.Based on the model of conventional photovoltaic(PV)and energy storage system(ESS),the mathematical optimization model of the system is proposed by taking the combined benefit of the building to the economy,society,and environment as the optimization objective,taking the near-zero energy consumption and carbon emission limitation of the building as the main constraints.The optimized operation strategy in this paper can give optimal results by making a trade-off between the users’costs and the combined benefits of the building.The efficiency and effectiveness of the proposed methods are verified by simulated experiments.

Base Station Energy Management in 5G Networks Using Wide Range Control Optimization

The traffic activity offifth generation(5G)networks demand for new energy management techniques that is dynamic deep and longer duration of sleep as compared to the fourth generation(4G)network technologies that demand always for varied control and data signalling based on control base station(CBS)and data base station(DBS).Hence,this paper discusses the energy management in wireless cellular networks using wide range of control for twice the reduction in energy conservation in non-standalone deployment of 5G network.As the new radio(NR)based 5G network is configured to transmit signal blocks for every 20 ms,the proposed algorithm implements withstanding capacity of on or off based energy switching,which in-turn operates in wide range control by carrying out reduced computational complexity.The proposed Wide range of control for base station in green cellular network using sleep mode for switch(WGCNS)algorithm toon and off the base station will work in heavy load with neighbouring base station.For reducing the overhead duration in air,heuristic versions of the algorithm are proposed at the base station.The algorithm operates based on the specification with suggested protocol-level to give best amount of energy savings.The proposed algorithm reduces 40%to 83%of residual energy based on the traffic pattern of the urban scenario.

Effective Energy Management Scheme by IMPC

The primary purpose of the Energy Management Scheme(EMS)is to monitor the energy fluctuations present in the load profile.In this paper,the improved model predictive controller is adopted for the EMS in the power system.Emperor Penguin Optimization(EPO)algorithm optimized Artificial Neural Network(ANN)with Model Predictive Control(MPC)scheme for accurate prediction of load and power forecasting at the time of preoptimizing EMS is presented.For the power generation,Renewable Energy Sources(RES)such as photo voltaic(PV)and wind turbine(WT)are utilized along with that the fuel cell is also presented in case of failure by the RES.Such a setup is connected with the grid and applies to the household appliances.In improved model predictive control(IMPC),the set of constraints for the powerflow in the system is optimized by the ANN,which is trained by EPO.Such a tuning based prediction model is presented in the IMPC technique.The proposed work is implemented in the MATLAB/Simulink platform.The energy management capability of the proposed system is analyzed for different atmospheric conditions.The total system cost,life cycle cost and annualized cost for IMPC are 48%,45%and 15%,respectively.From the performance analysis,the cost obtained by the proposed method is very low compared to that obtained by the existing techniques.

Determination of Effectiveness of Energy Management System in Buildings

Building Energy Management Systems(BEMS)are computer-based systems that aid in managing,controlling,and monitoring the building technical services and energy consumption by equipment used in the building.The effectiveness of BEMS is dependent upon numerous factors,among which the operational characteristics of the building and the BEMS control parameters also play an essential role.This research develops a user-driven simulation tool where users can input the building parameters and BEMS controls to determine the effectiveness of their BEMS.The simulation tool gives the user the flexibility to understand the potential energy savings by employing specific BEMS control and help in making intelligent decisions.The simulation is developed using Visual Basic Application(VBA)in Microsoft Excel,based on discrete-event Monte Carlo Simulation(MCS).The simulation works by initially calculating the energy required for space cooling and heating based on current building parameters input by the user in the model.Further,during the second simulation,the user selects all the BEMS controls and improved building envelope to determine the energy required for space cooling and heating during that case.The model compares the energy consumption from the first simulation and the second simulation.Then the simulation model will provide the rating of the effectiveness of BEMS on a continuous scale of 1 to 5(1 being poor effectiveness and 5 being excellent effectiveness of BEMS).This work is intended to facilitate building owner/energy managers to analyze the building energy performance concerning the efficacy of their energy management system.

Low Carbon Building Design Optimization Based on Intelligent Energy Management System

The construction of relevant standards for building carbon emission assessment in China has just started,and the quantitative analysis method and evaluation system are still imperfect,which hinders the development of low-carbon building design.Therefore,the use of intelligent energy management system is very necessary.The purpose of this paper is to explore the design optimization of low-carbon buildings based on intelligent energy management systems.Based on the proposed quantitative method of building carbon emission,this paper establishes the quota theoretical system of building carbon emission analysis,and develops the quota based carbon emission calculation software.Smart energy management system is a low-carbon energy-saving system based on the reference of large-scale building energy-saving system and combined with energy consumption.It provides a fast and effective calculation tool for the quantitative evaluation of carbon emission of construction projects,so as to realize the carbon emission control and optimization in the early stage of architectural design and construction.On this basis,the evaluation,analysis and calculation method of building structure based on carbon reduction target is proposed,combined with the carbon emission quota management standard proposed in this paper.Taking small high-rise residential buildings as an example,this paper compares and analyzes different building structural systems from the perspectives of structural performance,economy and carbon emission level.It provides a reference for the design and evaluation of low-carbon building structures.The smart energy management system collects user energy use parameters.It uses time period and time sequence to obtain a large amount of data for analysis and integration,which provides users with intuitive energy consumption data.Compared with the traditional architectural design method,the industrialized construction method can save 589.22 megajoules(MJ)per square meter.Based on 29270 megajoules(MJ)per ton of standard coal,the construction area of the case is about 8000 m2,and the energy saving of residential buildings is 161.04 tons of standard coal.This research is of great significance in reducing the carbon emission intensity of buildings.

Energy Management of Networked Smart Railway Stations Considering Regenerative Braking, Energy Storage System, and Photovoltaic Units

The networking of microgrids has received significant attention in the form of a smart grid.In this paper,a set of smart railway stations,which is assumed as microgrids,is connected together.It has been tried to manage the energy exchanged between the networked microgrids to reduce received energy from the utility grid.Also,the operational costs of stations under various conditions decrease by applying the proposed method.The smart railway stations are studied in the presence of photovoltaic(PV)units,energy storage systems(ESSs),and regenerative braking strategies.Studying regenerative braking is one of the essential contributions.Moreover,the stochastic behaviors of the ESS’s initial state of energy and the uncertainty of PV power generation are taken into account through a scenario-based method.The networked microgrid scheme of railway stations(based on coordinated operation and scheduling)and independent operation of railway stations are studied.The proposed method is applied to realistic case studies,including three stations of Line 3 of Tehran Urban and Suburban Railway Operation Company(TUSROC).The rolling stock is simulated in the MATLAB environment.Thus,the coordinated operation of networked microgrids and independent operation of railway stations are optimized in the GAMS environment utilizing mixed-integer linear programming(MILP).

Power-balancing Instantaneous Optimization Energy Management for a Novel Series-parallel Hybrid Electric Bus

Energy management（EM） is a core technique of hybrid electric bus（HEB） in order to advance fuel economy performance optimization and is unique for the corresponding configuration. There are existing algorithms of control strategy seldom take battery power management into account with international combustion engine power management. In this paper, a type of power-balancing instantaneous optimization（PBIO） energy management control strategy is proposed for a novel series-parallel hybrid electric bus. According to the characteristic of the novel series-parallel architecture, the switching boundary condition between series and parallel mode as well as the control rules of the power-balancing strategy are developed. The equivalent fuel model of battery is implemented and combined with the fuel of engine to constitute the objective function which is to minimize the fuel consumption at each sampled time and to coordinate the power distribution in real-time between the engine and battery. To validate the proposed strategy effective and reasonable, a forward model is built based on Matlab/Simulink for the simulation and the dSPACE autobox is applied to act as a controller for hardware in-the-loop integrated with bench test. Both the results of simulation and hardware-in-the-loop demonstrate that the proposed strategy not only enable to sustain the battery SOC within its operational range and keep the engine operation point locating the peak efficiency region, but also the fuel economy of series-parallel hybrid electric bus（SPHEB） dramatically advanced up to 30.73% via comparing with the prototype bus and a similar improvement for PBIO strategy relative to rule-based strategy, the reduction of fuel consumption is up to 12.38%. The proposed research ensures the algorithm of PBIO is real-time applicability, improves the efficiency of SPHEB system, as well as suite to complicated configuration perfectly.

Distributed Optimal Co-multi-microgrids Energy Management for Energy Internet

Unlike conventional power systems, the upcoming energy internet U+0028 EI U+0029 emphasizes comprehensive utilization of energy in the whole power system by coordinating multi-microgrids, which also brings new challenge for the energy management. To address this issue, this paper proposes a novel consensus-based distributed approach based on multi-agent framework to solve the energy management problem of the energy internet, which only requires local information exchange among neighboring agents. Correspondingly, two consensus algorithms are presented, one of which drives the incremental cost of each distributed generator U+0028 DG U+0029 converge to the state of the leader agent-energy router, and the other one is used to estimate the global power mismatch, which is a first-order average consensus algorithm modified by a correction term. In addition, in order to meet the supply-demand balance, an effective control strategy for the energy router is proposed to accurately calculate the power exchange between the microgrid and the main grid. Finally, simulation results within a 7-bus test system are provided to illustrate the effectiveness of the proposed approach. © 2014 Chinese Association of Automation.

An optimal energy management development for various configuration of plug-in and hybrid electric vehicle

Due to soaring fuel prices and environmental concerns, hybrid electric vehicle(HEV) technology attracts more attentions in last decade. Energy management system, configuration of HEV and traffic conditions are the main factors which affect HEV's fuel consumption, emission and performance. Therefore, optimal management of the energy components is a key element for the success of a HEV. An optimal energy management system is developed for HEV based on genetic algorithm. Then, different powertrain system component combinations effects are investigated in various driving cycles. HEV simulation results are compared for default rule-based, fuzzy and GA-fuzzy controllers by using ADVISOR. The results indicate the effectiveness of proposed optimal controller over real world driving cycles. Also, an optimal powertrain configuration to improve fuel consumption and emission efficiency is proposed for each driving condition. Finally, the effects of batteries in initial state of charge and hybridization factor are investigated on HEV performance to evaluate fuel consumption and emissions. Fuel consumption average reduction of about 14% is obtained for optimal configuration data in contrast to default configuration. Also results indicate that proposed controller has reduced emission of about 10% in various traffic conditions.

Energy Management Control StraEnergy Management Control Strategy for Renewable Energy System Based on Spotted Hyena Optimizertegy for Renewable Energy System Based on Spotted Hyena Optimizer

Hydrocarbons,carbon monoxide and other pollutants from the transportation sector harm human health in many ways.Fuel cell(FC)has been evolving rapidly over the past two decades due to its efficient mechanism to transform the chemical energy in hydrogen-rich compounds into electrical energy.The main drawback of the standalone FC is its slow dynamic response and its inability to supply rapid variations in the load demand.Therefore,adding energy storage systems is necessary.However,to manage and distribute the power-sharing among the hybrid proton exchange membrane(PEM)fuel cell(FC),battery storage(BS),and supercapacitor(SC),an energy management strategy(EMS)is essential.In this research work,an optimal EMS based on a spotted hyena optimizer(SHO)for hybrid PEM fuel cell/BS/SC is proposed.The main goal of an EMS is to improve the performance of hybrid FC/BS/SC and to reduce the amount of hydrogen consumption.To prove the superiority of the SHO method,the obtained results are compared with the chimp optimizer(CO),the artificial ecosystem-based optimizer(AEO),the seagull optimization algorithm(SOA),the sooty tern optimization algorithm(STOA),and the coyote optimization algorithm(COA).Two main metrics are used as a benchmark for the comparison:the minimum consumed hydrogen and the efficiency of the system.The main findings confirm that the minimum amount of hydrogen consumption and maximum efficiency are achieved by the proposed SHO based EMS.

Initiative Optimization Operation Strategy and Multi-objective Energy Management Method for Combined Cooling Heating and Power

This paper proposed an initiative optimization operation strategy and multi-objective energy management method for combined cooling heating and power U+0028 CCHP U+0029 with storage systems. Initially, the initiative optimization operation strategy of CCHP system in the cooling season, the heating season and the transition season was formulated. The energy management of CCHP system was optimized by the multi-objective optimization model with maximum daily energy efficiency, minimum daily carbon emissions and minimum daily operation cost based on the proposed initiative optimization operation strategy. Furthermore, the pareto optimal solution set was solved by using the niche particle swarm multi-objective optimization algorithm. Ultimately, the most satisfactory energy management scheme was obtained by using the technique for order preference by similarity to ideal solution U+0028 TOPSIS U+0029 method. A case study of CCHP system used in a hospital in the north of China validated the effectiveness of this method. The results showed that the satisfactory energy management scheme of CCHP system was obtained based on this initiative optimization operation strategy and multi-objective energy management method. The CCHP system has achieved better energy efficiency, environmental protection and economic benefits. © 2014 Chinese Association of Automation.

ENERGY MANAGEMENT STRATEGY FOR PARALLEL HYBRID ELECTRIC VEHICLES

Energy management strategy (EMS) is the core of the real-time controlalgorithm of the hybrid electric vehicle (HEV). A novel EMS using the logic threshold approach withincorporation of a stand-by optimization algorithm is proposed. The aim of it is to minimize theengine fuel consumption and maintain the battery state of charge (SOC) in its operation range, whilesatisfying the vehicle performance and drivability requirements. The hybrid powertrain bench testis carried out to collect data of the engine, motor and battery pack, which are used in the EMS tocontrol the powertrain. Computer simulation model of the HEV is established in the MATLAB/Simulinkenvironment according to the bench test results. Simulation results are presented for behaviors ofthe engine, motor and battery. The proposed EMS is implemented for a real parallel hybrid carcontrol system and validated by vehicle field tests.