Reinforcement Learning-Based Energy Management for Hybrid Power Systems:State-of-the-Art Survey,Review,and Perspectives

The new energy vehicle plays a crucial role in green transportation,and the energy management strategy of hybrid power systems is essential for ensuring energy-efficient driving.This paper presents a state-of-the-art survey and review of reinforcement learning-based energy management strategies for hybrid power systems.Additionally,it envisions the outlook for autonomous intelligent hybrid electric vehicles,with reinforcement learning as the foundational technology.First of all,to provide a macro view of historical development,the brief history of deep learning,reinforcement learning,and deep reinforcement learning is presented in the form of a timeline.Then,the comprehensive survey and review are conducted by collecting papers from mainstream academic databases.Enumerating most of the contributions based on three main directions—algorithm innovation,powertrain innovation,and environment innovation—provides an objective review of the research status.Finally,to advance the application of reinforcement learning in autonomous intelligent hybrid electric vehicles,future research plans positioned as“Alpha HEV”are envisioned,integrating Autopilot and energy-saving control.

Role of outdoor trees on pedestrian wind and thermal conditions around a pre-education building for sustainable energy management

Finding sustainable energy resources is essential to face the increasing energy demand.Trees are an important part of ancient architecture but are becoming rare in urban areas.Trees can control and tune the pedestrian-level wind velocity and thermal condition.In this study,a numerical investigation is employed to assess the role of trees planted in the windward direction of the building complex on the thermal and pedestrian wind velocity conditions around/inside a pre-education building located in the center of the complex.Compared to the previous studies(which considered only outside buildings),this work considers the effects of trees on microclimate change both inside/outside buildings.Effects of different parameters including the leaf area density and number of trees,number of rows,far-field velocity magnitude,and thermal condition around the main building are assessed.The results show that the flow velocity in the spacing between the first-row buildings is reduced by 30%-40% when the one-row trees with 2 m height are planted 15 m farther than the buildings.Furthermore,two rows of trees are more effective in higher velocities and reduce the maximum velocity by about 50%.The investigation shows that trees also could reduce the temperature by about 1℃around the building.

Regenerative Braking Energy Recovery System of Metro Train Based on Dual-Mode Power Management

In order to fully utilize the regenerative braking energy of metro trains and stabilize the metro DC traction busbar voltage,a hybrid regenerative braking energy recovery system with a dual-mode power management strategy is proposed.Firstly,the construction of the hybrid regenerative braking energy recovery system is explained.Then,based on the power demand of low-voltage load in metro stations,a dual-mode power management strategy is proposed to allocate the reference power of each system according to the different working conditions,and the control methods of each system are set.Finally,the correctness and effectiveness of the dual-mode strategy are verified through simulation,and the proposed braking energy utilization scheme is compared with other singleform utilization schemes.The results illustrate that the hybrid system with the dual-mode strategy can effectively recycle the regenerative braking energy of metro train and inhibit the busbar voltage fluctuation;the proposed braking energy utilization scheme has certain advantages on energy recovery and DC bus voltage stabilization compared with other single-form schemes;the proposed power management strategy can correctly allocate the reference power of each system with a lower construction cost.

Smart Energy Management System Using Machine Learning

Energy management is an inspiring domain in developing of renewable energy sources.However,the growth of decentralized energy production is revealing an increased complexity for power grid managers,inferring more quality and reliability to regulate electricity flows and less imbalance between electricity production and demand.The major objective of an energy management system is to achieve optimum energy procurement and utilization throughout the organization,minimize energy costs without affecting production,and minimize environmental effects.Modern energy management is an essential and complex subject because of the excessive consumption in residential buildings,which necessitates energy optimization and increased user comfort.To address the issue of energy management,many researchers have developed various frameworks;while the objective of each framework was to sustain a balance between user comfort and energy consumption,this problem hasn’t been fully solved because of how difficult it is to solve it.An inclusive and Intelligent Energy Management System(IEMS)aims to provide overall energy efficiency regarding increased power generation,increase flexibility,increase renewable generation systems,improve energy consumption,reduce carbon dioxide emissions,improve stability,and reduce energy costs.Machine Learning(ML)is an emerging approach that may be beneficial to predict energy efficiency in a better way with the assistance of the Internet of Energy(IoE)network.The IoE network is playing a vital role in the energy sector for collecting effective data and usage,resulting in smart resource management.In this research work,an IEMS is proposed for Smart Cities(SC)using the ML technique to better resolve the energy management problem.The proposed system minimized the energy consumption with its intelligent nature and provided better outcomes than the previous approaches in terms of 92.11% accuracy,and 7.89% miss-rate.

Reinforcement Learning Model for Energy System Management to Ensure Energy Efficiency and Comfort in Buildings

This article focuses on the challenges ofmodeling energy supply systems for buildings,encompassing both methods and tools for simulating thermal regimes and engineering systems within buildings.Enhancing the comfort of living or working in buildings often necessitates increased consumption of energy and material,such as for thermal upgrades,which consequently incurs additional economic costs.It is crucial to acknowledge that such improvements do not always lead to a decrease in total pollutant emissions,considering emissions across all stages of production and usage of energy and materials aimed at boosting energy efficiency and comfort in buildings.In addition,it explores the methods and mechanisms for modeling the operating modes of electric boilers used to collectively improve energy efficiency and indoor climatic conditions.Using the developed mathematical models,the study examines the dynamic states of building energy supply systems and provides recommendations for improving their efficiency.These dynamic models are executed in software environments such as MATLAB/Simscape and Python,where the component detailing schemes for various types of controllers are demonstrated.Additionally,controllers based on reinforcement learning(RL)displayed more adaptive load level management.These RL-based controllers can lower instantaneous power usage by up to 35%,reduce absolute deviations from a comfortable temperature nearly by half,and cut down energy consumption by approximately 1%while maintaining comfort.When the energy source produces a constant energy amount,the RL-based heat controllermore effectively maintains the temperature within the set range,preventing overheating.In conclusion,the introduced energydynamic building model and its software implementation offer a versatile tool for researchers,enabling the simulation of various energy supply systems to achieve optimal energy efficiency and indoor climate control in buildings.

Enhanced Deep Reinforcement Learning Strategy for Energy Management in Plug-in Hybrid Electric Vehicles with Entropy Regularization and Prioritized Experience Replay

Plug-in Hybrid Electric Vehicles(PHEVs)represent an innovative breed of transportation,harnessing diverse power sources for enhanced performance.Energy management strategies(EMSs)that coordinate and control different energy sources is a critical component of PHEV control technology,directly impacting overall vehicle performance.This study proposes an improved deep reinforcement learning(DRL)-based EMSthat optimizes realtime energy allocation and coordinates the operation of multiple power sources.Conventional DRL algorithms struggle to effectively explore all possible state-action combinations within high-dimensional state and action spaces.They often fail to strike an optimal balance between exploration and exploitation,and their assumption of a static environment limits their ability to adapt to changing conditions.Moreover,these algorithms suffer from low sample efficiency.Collectively,these factors contribute to convergence difficulties,low learning efficiency,and instability.To address these challenges,the Deep Deterministic Policy Gradient(DDPG)algorithm is enhanced using entropy regularization and a summation tree-based Prioritized Experience Replay(PER)method,aiming to improve exploration performance and learning efficiency from experience samples.Additionally,the correspondingMarkovDecision Process(MDP)is established.Finally,an EMSbased on the improvedDRLmodel is presented.Comparative simulation experiments are conducted against rule-based,optimization-based,andDRL-based EMSs.The proposed strategy exhibitsminimal deviation fromthe optimal solution obtained by the dynamic programming(DP)strategy that requires global information.In the typical driving scenarios based onWorld Light Vehicle Test Cycle(WLTC)and New European Driving Cycle(NEDC),the proposed method achieved a fuel consumption of 2698.65 g and an Equivalent Fuel Consumption(EFC)of 2696.77 g.Compared to the DP strategy baseline,the proposed method improved the fuel efficiency variances(FEV)by 18.13%,15.1%,and 8.37%over the Deep QNetwork(DQN),Double DRL(DDRL),and original DDPG methods,respectively.The observational outcomes demonstrate that the proposed EMS based on improved DRL framework possesses good real-time performance,stability,and reliability,effectively optimizing vehicle economy and fuel consumption.

A Predictive Energy Management Strategies for Mining Dump Trucks

The plug-in hybrid vehicles(PHEV)technology can effectively address the issues of poor dynamics and higher energy consumption commonly found in traditional mining dump trucks.Meanwhile,plug-in hybrid electric trucks can achieve excellent fuel economy through efficient energy management strategies(EMS).Therefore,a series hybrid system is constructed based on a 100-ton mining dump truck in this paper.And inspired by the dynamic programming(DP)algorithm,a predictive equivalent consumption minimization strategy(P-ECMS)based on the DP optimization result is proposed.Based on the optimal control manifold and the SOC reference trajectory obtained by the DP algorithm,the P-ECMS strategy performs real-time stage parameter optimization to obtain the optimal equivalent factor(EF).Finally,applying the equivalent consumption minimization strategy(ECMS)realizes real-time control.The simulation results show that the equivalent fuel consumption of the P-ECMS strategy under the experimentally collected mining cycle conditions is 150.8 L/100 km,which is 10.9%less than that of the common CDCS strategy(169.3 L/100 km),and achieves 99.47%of the fuel saving effect of the DP strategy(150 L/100 km).

Research on Smart Energy Monitoring and Management System Based on Digital Twin Technology

Smart energy monitoring and management system lays a foundation for the application and development of smart energy. However, in recent years, the work efficiency of smart energy development enterprises has generally been low, and there is an urgent need to improve the application efficiency, resilience and sustainability of smart energy monitoring and management system. Digital twin technology provides a data-centric solution to improve smart energy monitoring and management system, bringing an opportunity to transform passive infrastructure assets into data-centric systems. This paper expounds on the concept and key technologies of digital twin, and designs a smart energy monitoring and management system based on digital twin technology, which has dual significance for promoting the development of smart energy field and promoting the application of digital twin.

Survey on AI and Machine Learning Techniques for Microgrid Energy Management Systems

In the era of an energy revolution,grid decentralization has emerged as a viable solution to meet the increasing global energy demand by incorporating renewables at the distributed level.Microgrids are considered a driving component for accelerating grid decentralization.To optimally utilize the available resources and address potential challenges,there is a need to have an intelligent and reliable energy management system(EMS)for the microgrid.The artificial intelligence field has the potential to address the problems in EMS and can provide resilient,efficient,reliable,and scalable solutions.This paper presents an overview of existing conventional and AI-based techniques for energy management systems in microgrids.We analyze EMS methods for centralized,decentralized,and distributed microgrids separately.Then,we summarize machine learning techniques such as ANNs,federated learning,LSTMs,RNNs,and reinforcement learning for EMS objectives such as economic dispatch,optimal power flow,and scheduling.With the incorporation of AI,microgrids can achieve greater performance efficiency and more reliability for managing a large number of energy resources.However,challenges such as data privacy,security,scalability,explainability,etc.,need to be addressed.To conclude,the authors state the possible future research directions to explore AI-based EMS's potential in real-world applications.

Sustainable Energy Management with Traffic Prediction Strategy for Autonomous Vehicle Systems

Recent advancements of the intelligent transportation system(ITS)provide an effective way of improving the overall efficiency of the energy management strategy(EMSs)for autonomous vehicles(AVs).The use of AVs possesses many advantages such as congestion control,accident prevention,and etc.However,energy management and traffic flow prediction(TFP)still remains a challenging problem in AVs.The complexity and uncertainties of driving situations adequately affect the outcome of the designed EMSs.In this view,this paper presents novel sustainable energy management with traffic flow prediction strategy(SEM-TPS)for AVs.The SEM-TPS technique applies type II fuzzy logic system(T2FLS)energy management scheme to accomplish the desired engine torque based on distinct parameters.In addition,the membership functions of the T2FLS scheme are chosen optimally using the barnacles mating optimizer(BMO).For accurate TFP,the bidirectional gated recurrent neural network(Bi-GRNN)model is used in AVs.A comprehensive experimental validation process is performed and the results are inspected with respect to several evaluation metrics.The experimental outcomes highlighted the supreme performance of the SEM-TPS technique over the recent state of art approaches.

Energy Efficiency in Smart Grid: A Prospective Study on Energy Management Systems

The term Smart Grid has become a term to represent the benefits of a smart and sophisticated electrical grid, which can meet various social expectations related to sustainability and energy efficiency. The Smart Grid promises to enable a better power management for energy utilities and consumers, to provide the ability to integrate the power grid, to support the development of micro grids, and to involve citizens in energy management with higher levels of responsibility. However, this context comes with potential pitfalls, such as vulnerabilities to cyber-security and privacy risks. In this article, a prospective study about energy management, and exploring critical issues of modeling of energy management systems in a context Smart. Grid is presented along with background of energy management systems. An analysis of the demand response condition is also presented. Finally, the advantages and disadvantages of the implementation of energy management systems, and a comparison with the Brazilian electricity system are discussed.

Autonomous Multi-Factor Energy Flows Controller (AmEFC): Enhancing Renewable Energy Management with Intelligent Control Systems Integration

The transition to sustainable energy systems is one of the defining challenges of our time, necessitating innovations in how we generate, distribute, and manage electrical power. Micro-grids, as localized energy hubs, have emerged as a promising solution to integrate renewable energy sources, ensure energy security, and improve system resilience. The Autonomous multi-factor Energy Flow Controller (AmEFC) introduced in this paper addresses this need by offering a scalable, adaptable, and resilient framework for energy management within an on-grid micro-grid context. The urgency for such a system is predicated on the increasing volatility and unpredictability in energy landscapes, including fluctuating renewable outputs and changing load demands. To tackle these challenges, the AmEFC prototype incorporates a novel hierarchical control structure that leverages Renewable Energy Sources (RES), such as photovoltaic systems, wind turbines, and hydro pumps, alongside a sophisticated Battery Management System (BMS). Its prime objective is to maintain an uninterrupted power supply to critical loads, efficiently balance energy surplus through hydraulic storage, and ensure robust interaction with the main grid. A comprehensive Simulink model is developed to validate the functionality of the AmEFC, simulating real-world conditions and dynamic interactions among the components. The model assesses the system’s reliability in consistently powering critical loads and its efficacy in managing surplus energy. The inclusion of advanced predictive algorithms enables the AmEFC to anticipate energy production and consumption trends, integrating weather forecasting and inter-controller communication to optimize energy flow within and across micro-grids. This study’s significance lies in its potential to facilitate the seamless incorporation of RES into existing power systems, thus propelling the energy sector towards a more sustainable, autonomous, and resilient future. The results underscore the potential of such a system to revolutionize energy management practices and highlight the importance of smart controller systems in the era of smart grids.

A utility and easily fabricated dual-mode fiber film for efficient and comfortable thermal management

Nowadays, the global climate is constantly being destroyed and the fluctuations in ambient temperature are becoming more frequent. However, conventional single-mode thermal management strategies(heating or cooling) failed to resolve such dynamic temperature changes. Moreover, developing thermal management devices capable of accommodating these temperature variations while remaining simple to fabricate and durable has remained a formidable obstacle. To address these bottlenecks, we design and successfully fabricate a novel dual-mode hierarchical(DMH) composite film featuring a micronanofiber network structure, achieved through a straightforward two-step continuous electrospinning process. In cooling mode, it presents a high solar reflectivity of up to 97.7% and an excellent atmospheric transparent window(ATW) infrared emissivity of up to 98.9%. Noted that this DMH film could realize a cooling of 8.1 ℃ compared to the ambient temperature outdoors. In heating mode, it also exhibits a high solar absorptivity of 94.7% and heats up to 11.9 ℃ higher than black cotton fabric when utilized by individuals. In practical application scenarios, a seamless transition between efficient cooling and heating is achieved by simply flipping the film. More importantly, the DMH film combining the benefits of composites demonstrates portability, durability, and easy-cleaning, promising to achieve large-scale production and use of thermally managed textiles in the future. The energy savings offered by film applications provide a viable solution for the early realization of carbon neutrality.

Distributed Optimal Co-multi-microgrids Energy Management for Energy Internet

Unlike conventional power systems, the upcoming energy internet U+0028 EI U+0029 emphasizes comprehensive utilization of energy in the whole power system by coordinating multi-microgrids, which also brings new challenge for the energy management. To address this issue, this paper proposes a novel consensus-based distributed approach based on multi-agent framework to solve the energy management problem of the energy internet, which only requires local information exchange among neighboring agents. Correspondingly, two consensus algorithms are presented, one of which drives the incremental cost of each distributed generator U+0028 DG U+0029 converge to the state of the leader agent-energy router, and the other one is used to estimate the global power mismatch, which is a first-order average consensus algorithm modified by a correction term. In addition, in order to meet the supply-demand balance, an effective control strategy for the energy router is proposed to accurately calculate the power exchange between the microgrid and the main grid. Finally, simulation results within a 7-bus test system are provided to illustrate the effectiveness of the proposed approach. © 2014 Chinese Association of Automation.

Power-balancing Instantaneous Optimization Energy Management for a Novel Series-parallel Hybrid Electric Bus

Energy management（EM） is a core technique of hybrid electric bus（HEB） in order to advance fuel economy performance optimization and is unique for the corresponding configuration. There are existing algorithms of control strategy seldom take battery power management into account with international combustion engine power management. In this paper, a type of power-balancing instantaneous optimization（PBIO） energy management control strategy is proposed for a novel series-parallel hybrid electric bus. According to the characteristic of the novel series-parallel architecture, the switching boundary condition between series and parallel mode as well as the control rules of the power-balancing strategy are developed. The equivalent fuel model of battery is implemented and combined with the fuel of engine to constitute the objective function which is to minimize the fuel consumption at each sampled time and to coordinate the power distribution in real-time between the engine and battery. To validate the proposed strategy effective and reasonable, a forward model is built based on Matlab/Simulink for the simulation and the dSPACE autobox is applied to act as a controller for hardware in-the-loop integrated with bench test. Both the results of simulation and hardware-in-the-loop demonstrate that the proposed strategy not only enable to sustain the battery SOC within its operational range and keep the engine operation point locating the peak efficiency region, but also the fuel economy of series-parallel hybrid electric bus（SPHEB） dramatically advanced up to 30.73% via comparing with the prototype bus and a similar improvement for PBIO strategy relative to rule-based strategy, the reduction of fuel consumption is up to 12.38%. The proposed research ensures the algorithm of PBIO is real-time applicability, improves the efficiency of SPHEB system, as well as suite to complicated configuration perfectly.

ENERGY MANAGEMENT STRATEGY FOR PARALLEL HYBRID ELECTRIC VEHICLES

Energy management strategy (EMS) is the core of the real-time controlalgorithm of the hybrid electric vehicle (HEV). A novel EMS using the logic threshold approach withincorporation of a stand-by optimization algorithm is proposed. The aim of it is to minimize theengine fuel consumption and maintain the battery state of charge (SOC) in its operation range, whilesatisfying the vehicle performance and drivability requirements. The hybrid powertrain bench testis carried out to collect data of the engine, motor and battery pack, which are used in the EMS tocontrol the powertrain. Computer simulation model of the HEV is established in the MATLAB/Simulinkenvironment according to the bench test results. Simulation results are presented for behaviors ofthe engine, motor and battery. The proposed EMS is implemented for a real parallel hybrid carcontrol system and validated by vehicle field tests.

An optimal energy management development for various configuration of plug-in and hybrid electric vehicle

Due to soaring fuel prices and environmental concerns, hybrid electric vehicle(HEV) technology attracts more attentions in last decade. Energy management system, configuration of HEV and traffic conditions are the main factors which affect HEV's fuel consumption, emission and performance. Therefore, optimal management of the energy components is a key element for the success of a HEV. An optimal energy management system is developed for HEV based on genetic algorithm. Then, different powertrain system component combinations effects are investigated in various driving cycles. HEV simulation results are compared for default rule-based, fuzzy and GA-fuzzy controllers by using ADVISOR. The results indicate the effectiveness of proposed optimal controller over real world driving cycles. Also, an optimal powertrain configuration to improve fuel consumption and emission efficiency is proposed for each driving condition. Finally, the effects of batteries in initial state of charge and hybridization factor are investigated on HEV performance to evaluate fuel consumption and emissions. Fuel consumption average reduction of about 14% is obtained for optimal configuration data in contrast to default configuration. Also results indicate that proposed controller has reduced emission of about 10% in various traffic conditions.

Decentralized Control for Residential Energy Management of a Smart Users' Microgrid with Renewable Energy Exchange

This paper presents a decentralized control strategy for the scheduling of electrical energy activities of a microgrid composed of smart homes connected to a distributor and exchanging renewable energy produced by individually owned distributed energy resources. The scheduling problem is stated and solved with the aim of reducing the overall energy supply from the grid, by allowing users to exchange the surplus renewable energy and by optimally planning users' controllable loads. We assume that each smart home can both buy/sell energy from/to the grid taking into account time-varying non-linear pricing signals. Simultaneously, smart homes cooperate and may buy/sell locally harvested renewable energy from/to other smart homes. The resulting optimization problem is formulated as a non-convex non-linear programming problem with a coupling of decision variables in the constraints. The proposed solution is based on a novel heuristic iterative decentralized scheme algorithm that suitably extends the Alternating Direction Method of Multipliers to a non-convex and decentralized setting. We discuss the conditions that guarantee the convergence of the presented algorithm. Finally, the application of the proposed technique to a case study under several scenarios shows its effectiveness.

Initiative Optimization Operation Strategy and Multi-objective Energy Management Method for Combined Cooling Heating and Power

This paper proposed an initiative optimization operation strategy and multi-objective energy management method for combined cooling heating and power U+0028 CCHP U+0029 with storage systems. Initially, the initiative optimization operation strategy of CCHP system in the cooling season, the heating season and the transition season was formulated. The energy management of CCHP system was optimized by the multi-objective optimization model with maximum daily energy efficiency, minimum daily carbon emissions and minimum daily operation cost based on the proposed initiative optimization operation strategy. Furthermore, the pareto optimal solution set was solved by using the niche particle swarm multi-objective optimization algorithm. Ultimately, the most satisfactory energy management scheme was obtained by using the technique for order preference by similarity to ideal solution U+0028 TOPSIS U+0029 method. A case study of CCHP system used in a hospital in the north of China validated the effectiveness of this method. The results showed that the satisfactory energy management scheme of CCHP system was obtained based on this initiative optimization operation strategy and multi-objective energy management method. The CCHP system has achieved better energy efficiency, environmental protection and economic benefits. © 2014 Chinese Association of Automation.

Intermittent energy restriction in type 2 diabetes: A short discussion of medication management

AIM To discuss type 2 diabetes mellitus(T2DM) medication changes required during the popular 5:2 intermittent energy restriction(IER) diet. METHODS A search was conducted in MEDLINE, EMBASE, AMED, CINAHL and Cochrane library for original research articles investigating the use of very low calorie diets(VLCD) in people with T2 DM. The search terms used included "VLCD" or "very low energy diet" or "very low energy restriction" or "IER" or "intermittent fasting" or "calorie restriction" or "diabetes mellitus type 2" and "type 2 diabetes". Reference lists of selected articles were also screened for relevant publications. Only research articles written in English, which also included an explanation of medication changes were included. A recent pilot trial using the 5:2 IER method, conducted by our research group, will also be summarized.RESULTS A total of 8 studies were found that investigated the use of VLCD in T2 DM and discussed medication management. Overall these studies indicate that the use of a VLCD for people with T2 DM usually require the cessation of medication to prevent hypoglycemia. Therefore, the 5:2 IER method will also require medication changes, but as seen in our pilot trial, may not require total cessation of medication, rather a cessation on the 2 IER days only. CONCLUSION Guidelines outlined here can be used in the initial stages of a 2-d IER diet, but extensive blood glucose monitoring is still required to make the necessary individual reductions to medications in response to weight loss.