Reinforcement Learning-Based Energy Management for Hybrid Power Systems:State-of-the-Art Survey,Review,and Perspectives

The new energy vehicle plays a crucial role in green transportation,and the energy management strategy of hybrid power systems is essential for ensuring energy-efficient driving.This paper presents a state-of-the-art survey and review of reinforcement learning-based energy management strategies for hybrid power systems.Additionally,it envisions the outlook for autonomous intelligent hybrid electric vehicles,with reinforcement learning as the foundational technology.First of all,to provide a macro view of historical development,the brief history of deep learning,reinforcement learning,and deep reinforcement learning is presented in the form of a timeline.Then,the comprehensive survey and review are conducted by collecting papers from mainstream academic databases.Enumerating most of the contributions based on three main directions—algorithm innovation,powertrain innovation,and environment innovation—provides an objective review of the research status.Finally,to advance the application of reinforcement learning in autonomous intelligent hybrid electric vehicles,future research plans positioned as“Alpha HEV”are envisioned,integrating Autopilot and energy-saving control.

A Predictive Energy Management Strategies for Mining Dump Trucks

The plug-in hybrid vehicles(PHEV)technology can effectively address the issues of poor dynamics and higher energy consumption commonly found in traditional mining dump trucks.Meanwhile,plug-in hybrid electric trucks can achieve excellent fuel economy through efficient energy management strategies(EMS).Therefore,a series hybrid system is constructed based on a 100-ton mining dump truck in this paper.And inspired by the dynamic programming(DP)algorithm,a predictive equivalent consumption minimization strategy(P-ECMS)based on the DP optimization result is proposed.Based on the optimal control manifold and the SOC reference trajectory obtained by the DP algorithm,the P-ECMS strategy performs real-time stage parameter optimization to obtain the optimal equivalent factor(EF).Finally,applying the equivalent consumption minimization strategy(ECMS)realizes real-time control.The simulation results show that the equivalent fuel consumption of the P-ECMS strategy under the experimentally collected mining cycle conditions is 150.8 L/100 km,which is 10.9%less than that of the common CDCS strategy(169.3 L/100 km),and achieves 99.47%of the fuel saving effect of the DP strategy(150 L/100 km).

Enhanced Deep Reinforcement Learning Strategy for Energy Management in Plug-in Hybrid Electric Vehicles with Entropy Regularization and Prioritized Experience Replay

Plug-in Hybrid Electric Vehicles(PHEVs)represent an innovative breed of transportation,harnessing diverse power sources for enhanced performance.Energy management strategies(EMSs)that coordinate and control different energy sources is a critical component of PHEV control technology,directly impacting overall vehicle performance.This study proposes an improved deep reinforcement learning(DRL)-based EMSthat optimizes realtime energy allocation and coordinates the operation of multiple power sources.Conventional DRL algorithms struggle to effectively explore all possible state-action combinations within high-dimensional state and action spaces.They often fail to strike an optimal balance between exploration and exploitation,and their assumption of a static environment limits their ability to adapt to changing conditions.Moreover,these algorithms suffer from low sample efficiency.Collectively,these factors contribute to convergence difficulties,low learning efficiency,and instability.To address these challenges,the Deep Deterministic Policy Gradient(DDPG)algorithm is enhanced using entropy regularization and a summation tree-based Prioritized Experience Replay(PER)method,aiming to improve exploration performance and learning efficiency from experience samples.Additionally,the correspondingMarkovDecision Process(MDP)is established.Finally,an EMSbased on the improvedDRLmodel is presented.Comparative simulation experiments are conducted against rule-based,optimization-based,andDRL-based EMSs.The proposed strategy exhibitsminimal deviation fromthe optimal solution obtained by the dynamic programming(DP)strategy that requires global information.In the typical driving scenarios based onWorld Light Vehicle Test Cycle(WLTC)and New European Driving Cycle(NEDC),the proposed method achieved a fuel consumption of 2698.65 g and an Equivalent Fuel Consumption(EFC)of 2696.77 g.Compared to the DP strategy baseline,the proposed method improved the fuel efficiency variances(FEV)by 18.13%,15.1%,and 8.37%over the Deep QNetwork(DQN),Double DRL(DDRL),and original DDPG methods,respectively.The observational outcomes demonstrate that the proposed EMS based on improved DRL framework possesses good real-time performance,stability,and reliability,effectively optimizing vehicle economy and fuel consumption.

DEVELOPMENT OF THE ENERGY MANAGEMENT STRATEGY FOR PARALLEL HYBRID ELECTRIC URBAN BUSES

A novel parallel hybrid electrical urban bus （PHEUB） configuration consisting of an extra one-way clutch and an automatic mechanical transmission （AMT） is taken as the study subject. An energy management strategy combining a logic threshold approach and an instantaneous optimization algorithm is proposed for the investigated PHEUB. The objective of the energy management strategy is to achieve acceptable vehicle performance and drivability requirements while simultaneously maximizing the engine fuel consumption and maintaining the battery state of charge in its operation range at all times. Under the environment of Matlab/Simulink, a computer simulation model for the PHEUB is constructed by using the model building method combining theoretical analysis and bench test data. Simulation and experiment results for China Typical Bus Driving Schedule at Urban District （CTBDS_UD） are obtained, and the results indicate that the proposed control strategy not only controls the hybrid system efficiently but also improves the fuel economy significantly.

ENERGY MANAGEMENT STRATEGY FOR PARALLEL HYBRID ELECTRIC VEHICLES

Energy management strategy (EMS) is the core of the real-time controlalgorithm of the hybrid electric vehicle (HEV). A novel EMS using the logic threshold approach withincorporation of a stand-by optimization algorithm is proposed. The aim of it is to minimize theengine fuel consumption and maintain the battery state of charge (SOC) in its operation range, whilesatisfying the vehicle performance and drivability requirements. The hybrid powertrain bench testis carried out to collect data of the engine, motor and battery pack, which are used in the EMS tocontrol the powertrain. Computer simulation model of the HEV is established in the MATLAB/Simulinkenvironment according to the bench test results. Simulation results are presented for behaviors ofthe engine, motor and battery. The proposed EMS is implemented for a real parallel hybrid carcontrol system and validated by vehicle field tests.

Survey on AI and Machine Learning Techniques for Microgrid Energy Management Systems

In the era of an energy revolution,grid decentralization has emerged as a viable solution to meet the increasing global energy demand by incorporating renewables at the distributed level.Microgrids are considered a driving component for accelerating grid decentralization.To optimally utilize the available resources and address potential challenges,there is a need to have an intelligent and reliable energy management system(EMS)for the microgrid.The artificial intelligence field has the potential to address the problems in EMS and can provide resilient,efficient,reliable,and scalable solutions.This paper presents an overview of existing conventional and AI-based techniques for energy management systems in microgrids.We analyze EMS methods for centralized,decentralized,and distributed microgrids separately.Then,we summarize machine learning techniques such as ANNs,federated learning,LSTMs,RNNs,and reinforcement learning for EMS objectives such as economic dispatch,optimal power flow,and scheduling.With the incorporation of AI,microgrids can achieve greater performance efficiency and more reliability for managing a large number of energy resources.However,challenges such as data privacy,security,scalability,explainability,etc.,need to be addressed.To conclude,the authors state the possible future research directions to explore AI-based EMS's potential in real-world applications.

Simulation research of energy management strategy for dual mode plug-in hybrid electrical vehicles

In this paper, a plug-in hybrid electrical vehicle(PHEV) is taken as the research object, and its dynamic performance and economic performance are taken as the research goals. Battery charge-sustaining(CS) period is divided into power mode and economy mode. Energy management strategy designing methods of power mode and economy mode are proposed. Maximum velocity, acceleration performance and fuel consumption are simulated during the CS period in the AVL CRUISE simulation environment. The simulation results indicate that the maximum velocity and acceleration time of the power mode are better than those in the economy mode. Fuel consumption of the economy mode is better than that in the power mode. Fuel consumption of PHEV during the CS period is further improved by using the methods proposed in this paper, and this is meaningful for research and development of PHEV.

Initiative Optimization Operation Strategy and Multi-objective Energy Management Method for Combined Cooling Heating and Power

This paper proposed an initiative optimization operation strategy and multi-objective energy management method for combined cooling heating and power U+0028 CCHP U+0029 with storage systems. Initially, the initiative optimization operation strategy of CCHP system in the cooling season, the heating season and the transition season was formulated. The energy management of CCHP system was optimized by the multi-objective optimization model with maximum daily energy efficiency, minimum daily carbon emissions and minimum daily operation cost based on the proposed initiative optimization operation strategy. Furthermore, the pareto optimal solution set was solved by using the niche particle swarm multi-objective optimization algorithm. Ultimately, the most satisfactory energy management scheme was obtained by using the technique for order preference by similarity to ideal solution U+0028 TOPSIS U+0029 method. A case study of CCHP system used in a hospital in the north of China validated the effectiveness of this method. The results showed that the satisfactory energy management scheme of CCHP system was obtained based on this initiative optimization operation strategy and multi-objective energy management method. The CCHP system has achieved better energy efficiency, environmental protection and economic benefits. © 2014 Chinese Association of Automation.

Research on System Control and Energy Management Strategy of Flux-Modulated Compound-Structure Permanent Magnet Synchronous Machine

The flux-modulated compound-structure permanent magnet synchronous machine (CS-PMSM), composed of a brushless double rotor machine (DRM) and a conventional permanent magnet synchronous machine (PMSM), is a power split device for plug-in hybrid electric vehicles. In this paper, its operating principle and mathematical model are introduced. A modified current controller with decoupled state feedback is proposed and verified. The system control strategy is simulated in Matlab, and the feasibility of the control system is proven. To improve fuel economy, an energy management strategy based on fuzzy logic controller is proposed and evaluated by the Urban Dynamometer Driving Schedule (UDDS) drive cycle. The results show that the total energy consumption is similar to that of Prius 2012.

Energy management strategy based on dynamic programming with durability extension for fuel cell hybrid tramway

This paper proposes an energy management strategy for a fuel cell(FC)hybrid power system based on dynamic programming and state machine strategy,which takes into account the durability of the FC and the hydrogen consumption of the system.The strategy first uses the principle of dynamic programming to solve the optimal power distribution between the FC and supercapacitor(SC),and then uses the optimization results of dynamic programming to update the threshold values in each state of the finite state machine to realize real-time management of the output power of the FC and SC.An FC/SC hybrid tramway simulation platform is established based on RTLAB real-time simulator.The compared results verify that the proposed EMS can improve the durability of the FC,increase its working time in the high-efficiency range,effectively reduce the hydrogen consumption,and keep the state of charge in an ideal range.

Power-balancing Instantaneous Optimization Energy Management for a Novel Series-parallel Hybrid Electric Bus

Energy management（EM） is a core technique of hybrid electric bus（HEB） in order to advance fuel economy performance optimization and is unique for the corresponding configuration. There are existing algorithms of control strategy seldom take battery power management into account with international combustion engine power management. In this paper, a type of power-balancing instantaneous optimization（PBIO） energy management control strategy is proposed for a novel series-parallel hybrid electric bus. According to the characteristic of the novel series-parallel architecture, the switching boundary condition between series and parallel mode as well as the control rules of the power-balancing strategy are developed. The equivalent fuel model of battery is implemented and combined with the fuel of engine to constitute the objective function which is to minimize the fuel consumption at each sampled time and to coordinate the power distribution in real-time between the engine and battery. To validate the proposed strategy effective and reasonable, a forward model is built based on Matlab/Simulink for the simulation and the dSPACE autobox is applied to act as a controller for hardware in-the-loop integrated with bench test. Both the results of simulation and hardware-in-the-loop demonstrate that the proposed strategy not only enable to sustain the battery SOC within its operational range and keep the engine operation point locating the peak efficiency region, but also the fuel economy of series-parallel hybrid electric bus（SPHEB） dramatically advanced up to 30.73% via comparing with the prototype bus and a similar improvement for PBIO strategy relative to rule-based strategy, the reduction of fuel consumption is up to 12.38%. The proposed research ensures the algorithm of PBIO is real-time applicability, improves the efficiency of SPHEB system, as well as suite to complicated configuration perfectly.

Energy Management for a Residential Microgrid Using Wavelet Transform and Fuzzy Control Including a Vehicle-to-Grid System

This paper presents the design and implementation of an energy management system （EMS） with wavelet transform and fuzzy control for a residential micro-grid. The hybrid system in this paper consists of a wind turbine generator, photovoltaic （PV） panels, an electric vehicle （EV）, and a super capacitor （SC）, which is able to connect or disconnect to the main grid. The control strategy is responsible for compensating the difference between the generated power by the wind and solar generators and the demanded power by the loads. Wavelet transform decomposes the power difference into a smoothed component and a fast fluctuated component. The command approach used for fuzzy logic rules considers the state of charging （SOC） of EV, renewable production, and the load demand as parameters. Furthermore, the command rules are developed in order to ensure a reliable grid when taking into account the EV battery protection to decide the output power of the EV. The model of the hybrid system is developed in detail under Matlab/Simulink software environment.

An Advanced Energy Management System

A brief outline of Energy Management System (EMS) in China is given and OPEN-2000 EMS, combined with the novel technique of computer science, is introduced. The system has been developed by Power System Control Division of NARI and put into operation in the middle of 1998.

Energy Management of Networked Smart Railway Stations Considering Regenerative Braking, Energy Storage System, and Photovoltaic Units

The networking of microgrids has received significant attention in the form of a smart grid.In this paper,a set of smart railway stations,which is assumed as microgrids,is connected together.It has been tried to manage the energy exchanged between the networked microgrids to reduce received energy from the utility grid.Also,the operational costs of stations under various conditions decrease by applying the proposed method.The smart railway stations are studied in the presence of photovoltaic(PV)units,energy storage systems(ESSs),and regenerative braking strategies.Studying regenerative braking is one of the essential contributions.Moreover,the stochastic behaviors of the ESS’s initial state of energy and the uncertainty of PV power generation are taken into account through a scenario-based method.The networked microgrid scheme of railway stations(based on coordinated operation and scheduling)and independent operation of railway stations are studied.The proposed method is applied to realistic case studies,including three stations of Line 3 of Tehran Urban and Suburban Railway Operation Company(TUSROC).The rolling stock is simulated in the MATLAB environment.Thus,the coordinated operation of networked microgrids and independent operation of railway stations are optimized in the GAMS environment utilizing mixed-integer linear programming(MILP).

Research about Online Security Warning and Risk Assessment of Power Grid Based on Energy Management System

In this paper, online security warning and risk assessment of power grid are proposed, based on data from EMS （Energy Management System）, combined with information of real-time operation state, component status and external operating environment. It combines the two factors, contingency likelihood and severity, that determine system reliability, into risk indices on different loads and operation modes, which provide precise evaluation of the power grid＇s security performance. According to these indices, it can know the vulnerable area of the system and whether the normal operating mode or repair mode is over-limited or not, and provide decision-making support for dispatchers. Common cause outages and equipment-aging are considered in terms of the establishment of outage model. Multiple risk indices are defined in order to reflect the risk level of the power grid more comprehensively.

ASSESSMENT OF EFFECTIVE ENERGY RETROFIT STRATEGIES AND RELATED IMPACT ON INDOOR ENVIRONMENTAL QUALITY A case study of an elementary school in the State of Maryland

1.0.INTRODUCTION In the United States,K-12 school buildings spend more than$8 billion each year on energy-more than they spend on computers and textbooks combined[1].Most occupied older buildings demonstrate poor operational performance-for instance,more than 30 percent of schools were built before 1960,and 53 percent of public schools need to spend money on repairs,renovations,and modernization to ensure that the schools’onsite buildings are in good overall condition.And among public schools with permanent buildings,the environmental factors in the permanent buildings have been rated as unsatisfactory or very unsatisfactory in 5 to 17 percent of them[2].Indoor environment quality(IEQ)is one of the core issues addressed in the majority of sustainable building certification and design guidelines.Children spend a significant amount of time indoors in a school environment.And poor IEA can lead to sickness and absenteeism from school and eventually cause a decrease in student performance[3].Different building types and their IEQ characteristics can be partly attributed to building age and construction materials.[4]Improving the energy performance of school buildings could result in the direct benefit of reduced utility costs and improving the indoor quality could improve the students’learning environment.Research also suggests that aging school facilities and inefficient equipment have a detrimental effect on academic performance that can be reversed when schools are upgraded.[5]Several studies have linked better lighting,thermal comfort,and air quality to higher test scores.[6,7,8]Another benefit of improving the energy efficiency of education buildings is the potential increase in market value through recognition of green building practice and labeling,such as that of a LEED or net zero energy building.In addition,because of their educational function,high-performance or energy-efficient buildings are particularly valuable for institution clients and local government.More and more high-performance buildings,net zero energy buildings,and positive energy buildings serve as living laboratories for educational purposes.Currently,educational/institutional buildings represent the largest portion of NZE(net zero energy)projects.Educational buildings comprise 36 percent of net zero buildings according to a 2014 National New Building Institute report.Of the 58 net zero energy educational buildings,32 are used for kindergarten through grade 12(K-12),21 for higher education,and 5 for general education.[9]Finally,because educational buildings account for the third largest amount of building floor space in the United States,super energy-efficient educational buildings could provide other societal and economic benefits beyond the direct energy cost savings for three reasons:1)educational buildings offer high visibility that can influence community members and the next generation of citizens,2)success stories of the use of public funds that returns lower operating costs and healthier student learning environments provide documentation that can be used by others,and 3)this sector offers national and regional forums and associations to facilitate the transfer of best design and operational practices.

Cost-Based Research on Energy Management Strategy of Electric Vehicles Using Hybird Energy Storage System

This paper uses the minimization and weighted sum of battery capacity loss and energy consumption under driving cycles as objective functions to improve the economy of Electric Vehicles(EVs)with an hybrid energy storage system composed of power batteries and ultracapacitors.Furthermore,Dynamic Programming(DP)is employed to determine the objective function values under different weight coefficients,the comprehensive cost consisting of battery aging and power consumption costs,and the relationship between the hybrid power distribution.We also evaluate the real-time fuzzy Energy Management Strategy(EMS),fuzzy control strategies,and a strategy based on DP using the World Light vehicle Test Procedure(WLTP)driving cycle and a synthesis driving cycle derived from New European Driving Cycle(NEDC),WLTP,and Urban Dynamometer Driving Schedule(UDDS)as examples.Then,the proposed strategy is compared with the fuzzy control strategy and the strategy based on DP.Compared with fuzzy energy management strategy(namely FZY-EMS),the proposed EMS reduces the battery capacity loss and system energy consumption.The results demonstrate the effectiveness of the proposed EMS in improving EV economy.

一种基于EMS的广域备自投控制系统

常规的备自投装置只利用本地信息来制定相应的动作策略,动作逻辑固定,只能适应特定的运行方式。文中提出一种基于能量管理系统(EMS)的广域备自投控制系统,在控制中心建立广域备自投模型,实现实时的恢复控制。同时,系统拓展了备自投的功能,解决了单电源串行供电故障时的恢复供电问题。该系统在信息获取、控制策略形成和控制措施执行的整个过程中,综合利用全网信息,实现备自投与安全稳定装置的协调动作。另外,系统自动对控制策略和电网的安全性进行校核或调整,提高了备自投动作策略的安全性。工程应用表明了所开发系统的实用性。

基于统一支撑平台的EMS与WAMS集成方案

提出了基于统一平台支撑的能量管理系统(EMS)和广域测量系统(WAMS)集成方案,并结合WAMS的特点,分析了WAMS和EMS一体化系统对支撑平台的要求。在统一平台支撑下,WAMS和EMS共用历史数据服务器、Web服务器和人机工作站,只保留EMS和WAMS各自的数据采集服务器和应用服务器。这一方案在减少硬件投资的同时,可以最大限度地减轻维护工作量、提高使用方便性,有利于WAMS应用功能的进一步扩展。该方案的有效性得到了实际工程应用的验证。

基于EMS的电网规划分析系统的设计与实现

现有的能量管理系统(EMS)中已经存有当前电网的很多数据,可以加以利用。为此,设计并开发了一套基于EMS的规划分析系统。该系统采用IEC 61970标准,建立了一套基于多视图的图模一体化支撑平台,提供了方便、实用的分析工具。应用该系统可以充分利用在线运行的EMS的电网模型、数据和图形,方便地扩展出预想规划态电网模型,实现安全和经济等校核与分析功能。阐述了基于EMS的电网规划分析系统的体系结构、支持平台、应用功能以及需要解决的关键问题,并介绍了该系统在深圳供电局的实际应用情况。