Reinforcement Learning-Based Energy Management for Hybrid Power Systems:State-of-the-Art Survey,Review,and Perspectives

The new energy vehicle plays a crucial role in green transportation,and the energy management strategy of hybrid power systems is essential for ensuring energy-efficient driving.This paper presents a state-of-the-art survey and review of reinforcement learning-based energy management strategies for hybrid power systems.Additionally,it envisions the outlook for autonomous intelligent hybrid electric vehicles,with reinforcement learning as the foundational technology.First of all,to provide a macro view of historical development,the brief history of deep learning,reinforcement learning,and deep reinforcement learning is presented in the form of a timeline.Then,the comprehensive survey and review are conducted by collecting papers from mainstream academic databases.Enumerating most of the contributions based on three main directions—algorithm innovation,powertrain innovation,and environment innovation—provides an objective review of the research status.Finally,to advance the application of reinforcement learning in autonomous intelligent hybrid electric vehicles,future research plans positioned as“Alpha HEV”are envisioned,integrating Autopilot and energy-saving control.

Smart Energy Management System Using Machine Learning

Energy management is an inspiring domain in developing of renewable energy sources.However,the growth of decentralized energy production is revealing an increased complexity for power grid managers,inferring more quality and reliability to regulate electricity flows and less imbalance between electricity production and demand.The major objective of an energy management system is to achieve optimum energy procurement and utilization throughout the organization,minimize energy costs without affecting production,and minimize environmental effects.Modern energy management is an essential and complex subject because of the excessive consumption in residential buildings,which necessitates energy optimization and increased user comfort.To address the issue of energy management,many researchers have developed various frameworks;while the objective of each framework was to sustain a balance between user comfort and energy consumption,this problem hasn’t been fully solved because of how difficult it is to solve it.An inclusive and Intelligent Energy Management System(IEMS)aims to provide overall energy efficiency regarding increased power generation,increase flexibility,increase renewable generation systems,improve energy consumption,reduce carbon dioxide emissions,improve stability,and reduce energy costs.Machine Learning(ML)is an emerging approach that may be beneficial to predict energy efficiency in a better way with the assistance of the Internet of Energy(IoE)network.The IoE network is playing a vital role in the energy sector for collecting effective data and usage,resulting in smart resource management.In this research work,an IEMS is proposed for Smart Cities(SC)using the ML technique to better resolve the energy management problem.The proposed system minimized the energy consumption with its intelligent nature and provided better outcomes than the previous approaches in terms of 92.11% accuracy,and 7.89% miss-rate.

Reinforcement Learning Model for Energy System Management to Ensure Energy Efficiency and Comfort in Buildings

This article focuses on the challenges ofmodeling energy supply systems for buildings,encompassing both methods and tools for simulating thermal regimes and engineering systems within buildings.Enhancing the comfort of living or working in buildings often necessitates increased consumption of energy and material,such as for thermal upgrades,which consequently incurs additional economic costs.It is crucial to acknowledge that such improvements do not always lead to a decrease in total pollutant emissions,considering emissions across all stages of production and usage of energy and materials aimed at boosting energy efficiency and comfort in buildings.In addition,it explores the methods and mechanisms for modeling the operating modes of electric boilers used to collectively improve energy efficiency and indoor climatic conditions.Using the developed mathematical models,the study examines the dynamic states of building energy supply systems and provides recommendations for improving their efficiency.These dynamic models are executed in software environments such as MATLAB/Simscape and Python,where the component detailing schemes for various types of controllers are demonstrated.Additionally,controllers based on reinforcement learning(RL)displayed more adaptive load level management.These RL-based controllers can lower instantaneous power usage by up to 35%,reduce absolute deviations from a comfortable temperature nearly by half,and cut down energy consumption by approximately 1%while maintaining comfort.When the energy source produces a constant energy amount,the RL-based heat controllermore effectively maintains the temperature within the set range,preventing overheating.In conclusion,the introduced energydynamic building model and its software implementation offer a versatile tool for researchers,enabling the simulation of various energy supply systems to achieve optimal energy efficiency and indoor climate control in buildings.

Enhanced Deep Reinforcement Learning Strategy for Energy Management in Plug-in Hybrid Electric Vehicles with Entropy Regularization and Prioritized Experience Replay

Plug-in Hybrid Electric Vehicles(PHEVs)represent an innovative breed of transportation,harnessing diverse power sources for enhanced performance.Energy management strategies(EMSs)that coordinate and control different energy sources is a critical component of PHEV control technology,directly impacting overall vehicle performance.This study proposes an improved deep reinforcement learning(DRL)-based EMSthat optimizes realtime energy allocation and coordinates the operation of multiple power sources.Conventional DRL algorithms struggle to effectively explore all possible state-action combinations within high-dimensional state and action spaces.They often fail to strike an optimal balance between exploration and exploitation,and their assumption of a static environment limits their ability to adapt to changing conditions.Moreover,these algorithms suffer from low sample efficiency.Collectively,these factors contribute to convergence difficulties,low learning efficiency,and instability.To address these challenges,the Deep Deterministic Policy Gradient(DDPG)algorithm is enhanced using entropy regularization and a summation tree-based Prioritized Experience Replay(PER)method,aiming to improve exploration performance and learning efficiency from experience samples.Additionally,the correspondingMarkovDecision Process(MDP)is established.Finally,an EMSbased on the improvedDRLmodel is presented.Comparative simulation experiments are conducted against rule-based,optimization-based,andDRL-based EMSs.The proposed strategy exhibitsminimal deviation fromthe optimal solution obtained by the dynamic programming(DP)strategy that requires global information.In the typical driving scenarios based onWorld Light Vehicle Test Cycle(WLTC)and New European Driving Cycle(NEDC),the proposed method achieved a fuel consumption of 2698.65 g and an Equivalent Fuel Consumption(EFC)of 2696.77 g.Compared to the DP strategy baseline,the proposed method improved the fuel efficiency variances(FEV)by 18.13%,15.1%,and 8.37%over the Deep QNetwork(DQN),Double DRL(DDRL),and original DDPG methods,respectively.The observational outcomes demonstrate that the proposed EMS based on improved DRL framework possesses good real-time performance,stability,and reliability,effectively optimizing vehicle economy and fuel consumption.

A Predictive Energy Management Strategies for Mining Dump Trucks

The plug-in hybrid vehicles(PHEV)technology can effectively address the issues of poor dynamics and higher energy consumption commonly found in traditional mining dump trucks.Meanwhile,plug-in hybrid electric trucks can achieve excellent fuel economy through efficient energy management strategies(EMS).Therefore,a series hybrid system is constructed based on a 100-ton mining dump truck in this paper.And inspired by the dynamic programming(DP)algorithm,a predictive equivalent consumption minimization strategy(P-ECMS)based on the DP optimization result is proposed.Based on the optimal control manifold and the SOC reference trajectory obtained by the DP algorithm,the P-ECMS strategy performs real-time stage parameter optimization to obtain the optimal equivalent factor(EF).Finally,applying the equivalent consumption minimization strategy(ECMS)realizes real-time control.The simulation results show that the equivalent fuel consumption of the P-ECMS strategy under the experimentally collected mining cycle conditions is 150.8 L/100 km,which is 10.9%less than that of the common CDCS strategy(169.3 L/100 km),and achieves 99.47%of the fuel saving effect of the DP strategy(150 L/100 km).

Power-balancing Instantaneous Optimization Energy Management for a Novel Series-parallel Hybrid Electric Bus

Energy management（EM） is a core technique of hybrid electric bus（HEB） in order to advance fuel economy performance optimization and is unique for the corresponding configuration. There are existing algorithms of control strategy seldom take battery power management into account with international combustion engine power management. In this paper, a type of power-balancing instantaneous optimization（PBIO） energy management control strategy is proposed for a novel series-parallel hybrid electric bus. According to the characteristic of the novel series-parallel architecture, the switching boundary condition between series and parallel mode as well as the control rules of the power-balancing strategy are developed. The equivalent fuel model of battery is implemented and combined with the fuel of engine to constitute the objective function which is to minimize the fuel consumption at each sampled time and to coordinate the power distribution in real-time between the engine and battery. To validate the proposed strategy effective and reasonable, a forward model is built based on Matlab/Simulink for the simulation and the dSPACE autobox is applied to act as a controller for hardware in-the-loop integrated with bench test. Both the results of simulation and hardware-in-the-loop demonstrate that the proposed strategy not only enable to sustain the battery SOC within its operational range and keep the engine operation point locating the peak efficiency region, but also the fuel economy of series-parallel hybrid electric bus（SPHEB） dramatically advanced up to 30.73% via comparing with the prototype bus and a similar improvement for PBIO strategy relative to rule-based strategy, the reduction of fuel consumption is up to 12.38%. The proposed research ensures the algorithm of PBIO is real-time applicability, improves the efficiency of SPHEB system, as well as suite to complicated configuration perfectly.

Distributed Optimal Co-multi-microgrids Energy Management for Energy Internet

Unlike conventional power systems, the upcoming energy internet U+0028 EI U+0029 emphasizes comprehensive utilization of energy in the whole power system by coordinating multi-microgrids, which also brings new challenge for the energy management. To address this issue, this paper proposes a novel consensus-based distributed approach based on multi-agent framework to solve the energy management problem of the energy internet, which only requires local information exchange among neighboring agents. Correspondingly, two consensus algorithms are presented, one of which drives the incremental cost of each distributed generator U+0028 DG U+0029 converge to the state of the leader agent-energy router, and the other one is used to estimate the global power mismatch, which is a first-order average consensus algorithm modified by a correction term. In addition, in order to meet the supply-demand balance, an effective control strategy for the energy router is proposed to accurately calculate the power exchange between the microgrid and the main grid. Finally, simulation results within a 7-bus test system are provided to illustrate the effectiveness of the proposed approach. © 2014 Chinese Association of Automation.

An optimal energy management development for various configuration of plug-in and hybrid electric vehicle

Due to soaring fuel prices and environmental concerns, hybrid electric vehicle(HEV) technology attracts more attentions in last decade. Energy management system, configuration of HEV and traffic conditions are the main factors which affect HEV's fuel consumption, emission and performance. Therefore, optimal management of the energy components is a key element for the success of a HEV. An optimal energy management system is developed for HEV based on genetic algorithm. Then, different powertrain system component combinations effects are investigated in various driving cycles. HEV simulation results are compared for default rule-based, fuzzy and GA-fuzzy controllers by using ADVISOR. The results indicate the effectiveness of proposed optimal controller over real world driving cycles. Also, an optimal powertrain configuration to improve fuel consumption and emission efficiency is proposed for each driving condition. Finally, the effects of batteries in initial state of charge and hybridization factor are investigated on HEV performance to evaluate fuel consumption and emissions. Fuel consumption average reduction of about 14% is obtained for optimal configuration data in contrast to default configuration. Also results indicate that proposed controller has reduced emission of about 10% in various traffic conditions.

Energy Management Control StraEnergy Management Control Strategy for Renewable Energy System Based on Spotted Hyena Optimizertegy for Renewable Energy System Based on Spotted Hyena Optimizer

Hydrocarbons,carbon monoxide and other pollutants from the transportation sector harm human health in many ways.Fuel cell(FC)has been evolving rapidly over the past two decades due to its efficient mechanism to transform the chemical energy in hydrogen-rich compounds into electrical energy.The main drawback of the standalone FC is its slow dynamic response and its inability to supply rapid variations in the load demand.Therefore,adding energy storage systems is necessary.However,to manage and distribute the power-sharing among the hybrid proton exchange membrane(PEM)fuel cell(FC),battery storage(BS),and supercapacitor(SC),an energy management strategy(EMS)is essential.In this research work,an optimal EMS based on a spotted hyena optimizer(SHO)for hybrid PEM fuel cell/BS/SC is proposed.The main goal of an EMS is to improve the performance of hybrid FC/BS/SC and to reduce the amount of hydrogen consumption.To prove the superiority of the SHO method,the obtained results are compared with the chimp optimizer(CO),the artificial ecosystem-based optimizer(AEO),the seagull optimization algorithm(SOA),the sooty tern optimization algorithm(STOA),and the coyote optimization algorithm(COA).Two main metrics are used as a benchmark for the comparison:the minimum consumed hydrogen and the efficiency of the system.The main findings confirm that the minimum amount of hydrogen consumption and maximum efficiency are achieved by the proposed SHO based EMS.

Initiative Optimization Operation Strategy and Multi-objective Energy Management Method for Combined Cooling Heating and Power

This paper proposed an initiative optimization operation strategy and multi-objective energy management method for combined cooling heating and power U+0028 CCHP U+0029 with storage systems. Initially, the initiative optimization operation strategy of CCHP system in the cooling season, the heating season and the transition season was formulated. The energy management of CCHP system was optimized by the multi-objective optimization model with maximum daily energy efficiency, minimum daily carbon emissions and minimum daily operation cost based on the proposed initiative optimization operation strategy. Furthermore, the pareto optimal solution set was solved by using the niche particle swarm multi-objective optimization algorithm. Ultimately, the most satisfactory energy management scheme was obtained by using the technique for order preference by similarity to ideal solution U+0028 TOPSIS U+0029 method. A case study of CCHP system used in a hospital in the north of China validated the effectiveness of this method. The results showed that the satisfactory energy management scheme of CCHP system was obtained based on this initiative optimization operation strategy and multi-objective energy management method. The CCHP system has achieved better energy efficiency, environmental protection and economic benefits. © 2014 Chinese Association of Automation.

ENERGY MANAGEMENT STRATEGY FOR PARALLEL HYBRID ELECTRIC VEHICLES

Energy management strategy (EMS) is the core of the real-time controlalgorithm of the hybrid electric vehicle (HEV). A novel EMS using the logic threshold approach withincorporation of a stand-by optimization algorithm is proposed. The aim of it is to minimize theengine fuel consumption and maintain the battery state of charge (SOC) in its operation range, whilesatisfying the vehicle performance and drivability requirements. The hybrid powertrain bench testis carried out to collect data of the engine, motor and battery pack, which are used in the EMS tocontrol the powertrain. Computer simulation model of the HEV is established in the MATLAB/Simulinkenvironment according to the bench test results. Simulation results are presented for behaviors ofthe engine, motor and battery. The proposed EMS is implemented for a real parallel hybrid carcontrol system and validated by vehicle field tests.

Implications COVID-19 on Performance and Energy Management in the Production Electricity

The COVID-19 pandemic has directly impacted the electric power industry;the energy sector has experienced huge losses in electricity production.These losses have also affected the reliability of communication and employees’performance,hence destabilizing the electric power system.This article aims at achieving two objectives.First,analyzing the impact of the COVID-19 pandemic on the communication of performance(human error and human factors)and energy management in electricity production.Second,to develop a conceptual framework model to alleviate effects of the pandemic on the power sector and then improve energy management and human performance.This paper involves investigating the influence of the COVID19 pandemic on the global production of electricity in the first quarters of 2019 and 2020.A conceptual model was developed based on a case study.Additionally,to ensure reliability,a variable,namely COVID-19,was used as a moderator to examine the effects of the independent and dependent variables.The results show that scores for the internet of things(IoT)with awareness and communication(A&C)and workplace environment management were high with Cronbach’s alpha value of 0.87 for the IoT and 0.89 for A&C.These numbers are important indicators of factors that could affect performance and energy management and should not be overlooked by the top management.The results also indicate that the pandemic has had a direct effect on the electricity production sector,and the conceptual framework model revealed that COVID-19,as a moderator,has a direct effect on the variables that significantly affect the improvement of both energy management and employee performance.The case study’s results confirm the poor performance in power plant maintenance and operation,in which human error would increase especially in Iraqi power plants that have not yet adopted any internationally recognized standards for energy management.This paper contributes to the literature studying COVID-19’s impact on the electricity sector in two ways:first,by developing a model to assist the electricity production sector mitigating effects of the COVID-19 pandemic,and second,by providing a detailed investigation into the pandemic’s impact on the electricity sector’s global production.The findings are hoped to assist researchers and research centers in understanding the general and specific framework to manage the pandemic’s effects on electricity production.

Survey on AI and Machine Learning Techniques for Microgrid Energy Management Systems

In the era of an energy revolution,grid decentralization has emerged as a viable solution to meet the increasing global energy demand by incorporating renewables at the distributed level.Microgrids are considered a driving component for accelerating grid decentralization.To optimally utilize the available resources and address potential challenges,there is a need to have an intelligent and reliable energy management system(EMS)for the microgrid.The artificial intelligence field has the potential to address the problems in EMS and can provide resilient,efficient,reliable,and scalable solutions.This paper presents an overview of existing conventional and AI-based techniques for energy management systems in microgrids.We analyze EMS methods for centralized,decentralized,and distributed microgrids separately.Then,we summarize machine learning techniques such as ANNs,federated learning,LSTMs,RNNs,and reinforcement learning for EMS objectives such as economic dispatch,optimal power flow,and scheduling.With the incorporation of AI,microgrids can achieve greater performance efficiency and more reliability for managing a large number of energy resources.However,challenges such as data privacy,security,scalability,explainability,etc.,need to be addressed.To conclude,the authors state the possible future research directions to explore AI-based EMS's potential in real-world applications.

DEVELOPMENT OF THE ENERGY MANAGEMENT STRATEGY FOR PARALLEL HYBRID ELECTRIC URBAN BUSES

A novel parallel hybrid electrical urban bus （PHEUB） configuration consisting of an extra one-way clutch and an automatic mechanical transmission （AMT） is taken as the study subject. An energy management strategy combining a logic threshold approach and an instantaneous optimization algorithm is proposed for the investigated PHEUB. The objective of the energy management strategy is to achieve acceptable vehicle performance and drivability requirements while simultaneously maximizing the engine fuel consumption and maintaining the battery state of charge in its operation range at all times. Under the environment of Matlab/Simulink, a computer simulation model for the PHEUB is constructed by using the model building method combining theoretical analysis and bench test data. Simulation and experiment results for China Typical Bus Driving Schedule at Urban District （CTBDS_UD） are obtained, and the results indicate that the proposed control strategy not only controls the hybrid system efficiently but also improves the fuel economy significantly.

Distributionally robust optimization based chance-constrained energy management for hybrid energy powered cellular networks

Energy harvesting has been recognized as a promising technique with which to effectively reduce carbon emis-sions and electricity expenses of base stations.However,renewable energy is inherently stochastic and inter-mittent,imposing formidable challenges on reliably satisfying users'time-varying wireless traffic demands.In addition,the probability distribution of the renewable energy or users’wireless traffic demand is not always fully known in practice.In this paper,we minimize the total energy cost of a hybrid-energy-powered cellular network by jointly optimizing the energy sharing among base stations,the battery charging and discharging rates,and the energy purchased from the grid under the constraint of a limited battery size at each base station.In solving the formulated non-convex chance-constrained stochastic optimization problem,a new ambiguity set is built to characterize the uncertainties in the renewable energy and wireless traffic demands according to interval sets of the mean and covariance.Using this ambiguity set,the original optimization problem is transformed into a more tractable second-order cone programming problem by exploiting the distributionally robust optimization approach.Furthermore,a low-complexity distributionally robust chance-constrained energy management algo-rithm,which requires only interval sets of the mean and covariance of stochastic parameters,is proposed.The results of extensive simulation are presented to demonstrate that the proposed algorithm outperforms existing methods in terms of the computational complexity,energy cost,and reliability.

Simulation research of energy management strategy for dual mode plug-in hybrid electrical vehicles

In this paper, a plug-in hybrid electrical vehicle(PHEV) is taken as the research object, and its dynamic performance and economic performance are taken as the research goals. Battery charge-sustaining(CS) period is divided into power mode and economy mode. Energy management strategy designing methods of power mode and economy mode are proposed. Maximum velocity, acceleration performance and fuel consumption are simulated during the CS period in the AVL CRUISE simulation environment. The simulation results indicate that the maximum velocity and acceleration time of the power mode are better than those in the economy mode. Fuel consumption of the economy mode is better than that in the power mode. Fuel consumption of PHEV during the CS period is further improved by using the methods proposed in this paper, and this is meaningful for research and development of PHEV.

Fuzzy Adaptive Filtering-Based Energy Management for Hybrid Energy Storage System

Regarding the problem of the short driving distance of pure electric vehicles,a battery,super-capacitor,and DC/DC converter are combined to form a hybrid energy storage system(HESS).A fuzzy adaptive filtering-based energy management strategy(FAFBEMS)is proposed to allocate the required power of the vehicle.Firstly,the state of charge(SOC)of the super-capacitor is limited according to the driving/braking mode of the vehicle to ensure that it is in a suitable working state,and fuzzy rules are designed to adaptively adjust the filtering time constant,to realize reasonable power allocation.Then,the positive and negative power are determined,and the average power of driving/braking is calculated so as to limit the power amplitude to protect the battery.To verify the proposed FAFBEMS strategy for HESS,simulations are performed under the UDDS(Urban Dynamometer Driving Schedule)driving cycle.The results show that the FAFBEMS strategy can effectively reduce the current amplitude of the battery,and the final SOC of the battery and super-capacitor is optimized to varying degrees.The energy consumption is 7.8%less than that of the rule-based energy management strategy,10.9%less than that of the fuzzy control energy management strategy,and 13.1%less than that of the filtering-based energy management strategy,which verifies the effectiveness of the FAFBEMS strategy.

Energy Management for a Residential Microgrid Using Wavelet Transform and Fuzzy Control Including a Vehicle-to-Grid System

This paper presents the design and implementation of an energy management system （EMS） with wavelet transform and fuzzy control for a residential micro-grid. The hybrid system in this paper consists of a wind turbine generator, photovoltaic （PV） panels, an electric vehicle （EV）, and a super capacitor （SC）, which is able to connect or disconnect to the main grid. The control strategy is responsible for compensating the difference between the generated power by the wind and solar generators and the demanded power by the loads. Wavelet transform decomposes the power difference into a smoothed component and a fast fluctuated component. The command approach used for fuzzy logic rules considers the state of charging （SOC） of EV, renewable production, and the load demand as parameters. Furthermore, the command rules are developed in order to ensure a reliable grid when taking into account the EV battery protection to decide the output power of the EV. The model of the hybrid system is developed in detail under Matlab/Simulink software environment.

Reinforcement Learning-Based Electric Vehicles Energy Management Strategy with Battery Thermal Model

The promotion of electric vehicles(EVs)is restricted due to their short cruising range.It is desirable to design an effective energy management strategy to improve their energy efficiency.Most existing work concerning energy management strategies focused on hybrids rather than the EVs.The work focusing on the energy management strategy for EVs mainly uses the traditional optimization strategies,thereby limiting the advantages of energy economy.To this end,a novel energy management strategy that considered the impact of battery thermal effects was proposed with the help of reinforcement learning.The main idea was to first analyze the energy flow path of EVs,further formulize the energy management as an optimization problem,and finally propose an online strategy based on reinforcement learning to obtain the optimal strategy.Additionally,extensive simulation results have demonstrated that our strategy reduces energy consumption by at least 27.4%compared to the existing methods.

Study the Role of Maritime Energy Management in Achieving the United Nations Sustainable Development Goals (UN SDGs), in particular, Goal 7, in Oman Maritime Community

Sustainability is a concept for achieving integration and balance in development to preserve and use resources properly by providing new solutions to overcome structural,social and economic misconceptions to prevent further degradation and waste of natural resources and create a better future for human societies.Sustainable development can be defined as a strategy for the correct use of resources,by modifying and redefining consumption patterns and avoiding one-sided development approaches.Therefore,sustainable development is defined as a long-term and unlimited solution for aligning different concepts(social,economic,and environmental),seemingly independent but related and influencing each other.This paper examines the challenges and capacities of the Oman Maritime Community and Maritime Energy Management's role in terms of sustainable development.To analyze this issue,the role of key players in Oman's maritime industry,such as the Oman Fisheries Company and the Oman Shipping Company,has been studied with the five main indicators of sustainable development goals,including people,the planet,peace,welfare,and partnership.