Enhanced Deep Reinforcement Learning Strategy for Energy Management in Plug-in Hybrid Electric Vehicles with Entropy Regularization and Prioritized Experience Replay

Plug-in Hybrid Electric Vehicles(PHEVs)represent an innovative breed of transportation,harnessing diverse power sources for enhanced performance.Energy management strategies(EMSs)that coordinate and control different energy sources is a critical component of PHEV control technology,directly impacting overall vehicle performance.This study proposes an improved deep reinforcement learning(DRL)-based EMSthat optimizes realtime energy allocation and coordinates the operation of multiple power sources.Conventional DRL algorithms struggle to effectively explore all possible state-action combinations within high-dimensional state and action spaces.They often fail to strike an optimal balance between exploration and exploitation,and their assumption of a static environment limits their ability to adapt to changing conditions.Moreover,these algorithms suffer from low sample efficiency.Collectively,these factors contribute to convergence difficulties,low learning efficiency,and instability.To address these challenges,the Deep Deterministic Policy Gradient(DDPG)algorithm is enhanced using entropy regularization and a summation tree-based Prioritized Experience Replay(PER)method,aiming to improve exploration performance and learning efficiency from experience samples.Additionally,the correspondingMarkovDecision Process(MDP)is established.Finally,an EMSbased on the improvedDRLmodel is presented.Comparative simulation experiments are conducted against rule-based,optimization-based,andDRL-based EMSs.The proposed strategy exhibitsminimal deviation fromthe optimal solution obtained by the dynamic programming(DP)strategy that requires global information.In the typical driving scenarios based onWorld Light Vehicle Test Cycle(WLTC)and New European Driving Cycle(NEDC),the proposed method achieved a fuel consumption of 2698.65 g and an Equivalent Fuel Consumption(EFC)of 2696.77 g.Compared to the DP strategy baseline,the proposed method improved the fuel efficiency variances(FEV)by 18.13%,15.1%,and 8.37%over the Deep QNetwork(DQN),Double DRL(DDRL),and original DDPG methods,respectively.The observational outcomes demonstrate that the proposed EMS based on improved DRL framework possesses good real-time performance,stability,and reliability,effectively optimizing vehicle economy and fuel consumption.

Reinforcement Learning-Based Energy Management for Hybrid Power Systems:State-of-the-Art Survey,Review,and Perspectives

The new energy vehicle plays a crucial role in green transportation,and the energy management strategy of hybrid power systems is essential for ensuring energy-efficient driving.This paper presents a state-of-the-art survey and review of reinforcement learning-based energy management strategies for hybrid power systems.Additionally,it envisions the outlook for autonomous intelligent hybrid electric vehicles,with reinforcement learning as the foundational technology.First of all,to provide a macro view of historical development,the brief history of deep learning,reinforcement learning,and deep reinforcement learning is presented in the form of a timeline.Then,the comprehensive survey and review are conducted by collecting papers from mainstream academic databases.Enumerating most of the contributions based on three main directions—algorithm innovation,powertrain innovation,and environment innovation—provides an objective review of the research status.Finally,to advance the application of reinforcement learning in autonomous intelligent hybrid electric vehicles,future research plans positioned as“Alpha HEV”are envisioned,integrating Autopilot and energy-saving control.

Simulation research of energy management strategy for dual mode plug-in hybrid electrical vehicles

In this paper, a plug-in hybrid electrical vehicle(PHEV) is taken as the research object, and its dynamic performance and economic performance are taken as the research goals. Battery charge-sustaining(CS) period is divided into power mode and economy mode. Energy management strategy designing methods of power mode and economy mode are proposed. Maximum velocity, acceleration performance and fuel consumption are simulated during the CS period in the AVL CRUISE simulation environment. The simulation results indicate that the maximum velocity and acceleration time of the power mode are better than those in the economy mode. Fuel consumption of the economy mode is better than that in the power mode. Fuel consumption of PHEV during the CS period is further improved by using the methods proposed in this paper, and this is meaningful for research and development of PHEV.

ENERGY MANAGEMENT STRATEGY FOR PARALLEL HYBRID ELECTRIC VEHICLES

Energy management strategy (EMS) is the core of the real-time controlalgorithm of the hybrid electric vehicle (HEV). A novel EMS using the logic threshold approach withincorporation of a stand-by optimization algorithm is proposed. The aim of it is to minimize theengine fuel consumption and maintain the battery state of charge (SOC) in its operation range, whilesatisfying the vehicle performance and drivability requirements. The hybrid powertrain bench testis carried out to collect data of the engine, motor and battery pack, which are used in the EMS tocontrol the powertrain. Computer simulation model of the HEV is established in the MATLAB/Simulinkenvironment according to the bench test results. Simulation results are presented for behaviors ofthe engine, motor and battery. The proposed EMS is implemented for a real parallel hybrid carcontrol system and validated by vehicle field tests.

A Predictive Energy Management Strategies for Mining Dump Trucks

The plug-in hybrid vehicles(PHEV)technology can effectively address the issues of poor dynamics and higher energy consumption commonly found in traditional mining dump trucks.Meanwhile,plug-in hybrid electric trucks can achieve excellent fuel economy through efficient energy management strategies(EMS).Therefore,a series hybrid system is constructed based on a 100-ton mining dump truck in this paper.And inspired by the dynamic programming(DP)algorithm,a predictive equivalent consumption minimization strategy(P-ECMS)based on the DP optimization result is proposed.Based on the optimal control manifold and the SOC reference trajectory obtained by the DP algorithm,the P-ECMS strategy performs real-time stage parameter optimization to obtain the optimal equivalent factor(EF).Finally,applying the equivalent consumption minimization strategy(ECMS)realizes real-time control.The simulation results show that the equivalent fuel consumption of the P-ECMS strategy under the experimentally collected mining cycle conditions is 150.8 L/100 km,which is 10.9%less than that of the common CDCS strategy(169.3 L/100 km),and achieves 99.47%of the fuel saving effect of the DP strategy(150 L/100 km).

An optimal energy management development for various configuration of plug-in and hybrid electric vehicle

Due to soaring fuel prices and environmental concerns, hybrid electric vehicle(HEV) technology attracts more attentions in last decade. Energy management system, configuration of HEV and traffic conditions are the main factors which affect HEV's fuel consumption, emission and performance. Therefore, optimal management of the energy components is a key element for the success of a HEV. An optimal energy management system is developed for HEV based on genetic algorithm. Then, different powertrain system component combinations effects are investigated in various driving cycles. HEV simulation results are compared for default rule-based, fuzzy and GA-fuzzy controllers by using ADVISOR. The results indicate the effectiveness of proposed optimal controller over real world driving cycles. Also, an optimal powertrain configuration to improve fuel consumption and emission efficiency is proposed for each driving condition. Finally, the effects of batteries in initial state of charge and hybridization factor are investigated on HEV performance to evaluate fuel consumption and emissions. Fuel consumption average reduction of about 14% is obtained for optimal configuration data in contrast to default configuration. Also results indicate that proposed controller has reduced emission of about 10% in various traffic conditions.

DEVELOPMENT OF THE ENERGY MANAGEMENT STRATEGY FOR PARALLEL HYBRID ELECTRIC URBAN BUSES

A novel parallel hybrid electrical urban bus （PHEUB） configuration consisting of an extra one-way clutch and an automatic mechanical transmission （AMT） is taken as the study subject. An energy management strategy combining a logic threshold approach and an instantaneous optimization algorithm is proposed for the investigated PHEUB. The objective of the energy management strategy is to achieve acceptable vehicle performance and drivability requirements while simultaneously maximizing the engine fuel consumption and maintaining the battery state of charge in its operation range at all times. Under the environment of Matlab/Simulink, a computer simulation model for the PHEUB is constructed by using the model building method combining theoretical analysis and bench test data. Simulation and experiment results for China Typical Bus Driving Schedule at Urban District （CTBDS_UD） are obtained, and the results indicate that the proposed control strategy not only controls the hybrid system efficiently but also improves the fuel economy significantly.

A Novel Method for the Application of the ECMS(Equivalent Consumption Minimization Strategy)to Reduce Hydrogen Consumption in Fuel Cell Hybrid Electric Vehicles

Fuel cell hybrid electric vehicles are currently being considered as ideal means to solve the energy crisis and global warming in today’s society.In this context,this paper proposes a method to solve the problem related to the dependence of the so-called optimal equivalent factor(determined in the framework of the equivalent consumption minimum strategy-ECMS)on the working conditions.The simulation results show that under typical conditions(some representative cities being considered),the proposed strategy can maintain the power balance;for different initial battery’s states of charge(SOC),after the SOC stabilizes,the fuel consumption is 5.25 L/100 km.

Optimizing hybrid electric vehicle coupling organic Rankine cycle energy management strategy via deep reinforcement learning

Trucks consume a lot of energy. Hybrid technology maintains a long range while realizing energy savings. Hybrid is therefore an effective energy-saving technology for trucks. Recovery of engine waste heat through the organic Rankine cycle further enhances engine efficiency and provides effective thermal management. However, the powertrain greatly increases the complexity of energy management system. In order to design an energy management system with high efficiency and robustness, this study proposes a deep reinforcement learning embedded rule-based energy management system. This method optimises the key parameters of the rule-based energy management system by inserting deep reinforcement learning into it. Therefore, this scheme combines the good optimization effect of deep reinforcement learning and the excellent robustness of rule. In order to verify the feasibility of this scheme, this study builds the system dynamic model and carries out a simulation study. Subsequently, a hybrid powertrain semi physical experimental bench was constructed and a rapid control prototype experimental study was carried out. The simulation results show that the deep reinforcement learning embedded rule-based energy management system can reduce the energy consumption by 4.31 % compared with the rule-based energy management system under the C-WTVC driving cycle. In addition, energy saving and safe operation can also be achieved under other unfamiliar untrained driving cycles. The rapid control prototype experimental study shows that the deep reinforcement learning embedded rule-based energy management system has good agreement in experiment and simulation, which demonstrates the potential for real vehicle engineering applications and promotes the engineering application of deep reinforcement learning.

Power-balancing Instantaneous Optimization Energy Management for a Novel Series-parallel Hybrid Electric Bus

Energy management（EM） is a core technique of hybrid electric bus（HEB） in order to advance fuel economy performance optimization and is unique for the corresponding configuration. There are existing algorithms of control strategy seldom take battery power management into account with international combustion engine power management. In this paper, a type of power-balancing instantaneous optimization（PBIO） energy management control strategy is proposed for a novel series-parallel hybrid electric bus. According to the characteristic of the novel series-parallel architecture, the switching boundary condition between series and parallel mode as well as the control rules of the power-balancing strategy are developed. The equivalent fuel model of battery is implemented and combined with the fuel of engine to constitute the objective function which is to minimize the fuel consumption at each sampled time and to coordinate the power distribution in real-time between the engine and battery. To validate the proposed strategy effective and reasonable, a forward model is built based on Matlab/Simulink for the simulation and the dSPACE autobox is applied to act as a controller for hardware in-the-loop integrated with bench test. Both the results of simulation and hardware-in-the-loop demonstrate that the proposed strategy not only enable to sustain the battery SOC within its operational range and keep the engine operation point locating the peak efficiency region, but also the fuel economy of series-parallel hybrid electric bus（SPHEB） dramatically advanced up to 30.73% via comparing with the prototype bus and a similar improvement for PBIO strategy relative to rule-based strategy, the reduction of fuel consumption is up to 12.38%. The proposed research ensures the algorithm of PBIO is real-time applicability, improves the efficiency of SPHEB system, as well as suite to complicated configuration perfectly.

Energy Control of Plug-In Hybrid Electric Vehicles Using Model Predictive Control With Route Preview

The paper proposes an adoption of slope,elevation,speed and route distance preview to achieve optimal energymanagement of plug-in hybrid electric vehicles(PHEVs).Theapproach is to identify route features from historical and real-time traffic data,in which information fusion model and trafficprediction model are used to improve the information accuracy.Then,dynamic programming combined with equivalent con-sumption minimization strategy is used to compute an optimalsolution for real-time energy management.The solution is thereference for PHEV energy management control along the route.To improve the system's ability of handling changing situation,the study further explores predictive control model in the real-time control of the energy.A simulation is performed to modelPHEV under above energy control strategy with route preview.The results show that the average fuel consumption of PHEValong the previewed route with model predictive control(MPC)strategy can be reduced compared with optimal strategy andbase control strategy.

Fuzzy Adaptive Filtering-Based Energy Management for Hybrid Energy Storage System

Regarding the problem of the short driving distance of pure electric vehicles,a battery,super-capacitor,and DC/DC converter are combined to form a hybrid energy storage system(HESS).A fuzzy adaptive filtering-based energy management strategy(FAFBEMS)is proposed to allocate the required power of the vehicle.Firstly,the state of charge(SOC)of the super-capacitor is limited according to the driving/braking mode of the vehicle to ensure that it is in a suitable working state,and fuzzy rules are designed to adaptively adjust the filtering time constant,to realize reasonable power allocation.Then,the positive and negative power are determined,and the average power of driving/braking is calculated so as to limit the power amplitude to protect the battery.To verify the proposed FAFBEMS strategy for HESS,simulations are performed under the UDDS(Urban Dynamometer Driving Schedule)driving cycle.The results show that the FAFBEMS strategy can effectively reduce the current amplitude of the battery,and the final SOC of the battery and super-capacitor is optimized to varying degrees.The energy consumption is 7.8%less than that of the rule-based energy management strategy,10.9%less than that of the fuzzy control energy management strategy,and 13.1%less than that of the filtering-based energy management strategy,which verifies the effectiveness of the FAFBEMS strategy.

Implementation of Radial Basis Function Artificial Neural Network into an Adaptive Equivalent Consumption Minimization Strategy for Optimized Control of a Hybrid Electric Vehicle

Continued increases in the emission of greenhouse gases by passenger vehicles have accelerated the production of hybrid electric vehicles. With this increase in production, there has been a parallel demand for continuously improving strategies of hybrid electric vehicle control. The goal of an ideal control strategy is to maximize fuel economy while minimizing emissions. Methods exist by which the globally optimal control strategy may be found. However, these methods are not applicable in real-world driving applications since these methods require a priori knowledge of the upcoming drive cycle. Real-time control strategies use the global optimal as a benchmark against which performance can be evaluated. The goal of this work is to use a previously defined strategy that has been shown to closely approximate the global optimal and implement a radial basis function (RBF) artificial neural network (ANN) that dynamically adapts the strategy based on past driving conditions. The strategy used is the Equivalent Consumption Minimization Strategy (ECMS), which uses an equivalence factor to define the control strategy and the power train component torque split. An equivalence factor that is optimal for a single drive cycle can be found offline with a priori knowledge of the drive cycle. The RBF-ANN is used to dynamically update the equivalence factor by examining a past time window of driving characteristics. A total of 30 sets of training data (drive cycles) are used to train the RBF-ANN. For the majority of drive cycles examined, the RBF-ANN implementation is shown to produce fuel economy values that are within ±2.5% of the fuel economy obtained with the optimal equivalence factor. The advantage of the RBF-ANN is that it does not require a priori drive cycle knowledge and is able to be implemented in real-time while meeting or exceeding the performance of the optimal ECMS. Recommendations are made on how the RBF-ANN could be improved to produce better results across a greater array of driving conditions.

Closed-form solution to the dynamic programming for a heavy-duty parallel hybrid vehicle energy management

Dynamic programming(DP)is frequently used to obtain the optimal solution to the hybrid electric vehicle(HEV)energy management.The trade-off between the accuracy and the computational effort is the biggest problem for the DP method.The closed-form solution to the DP is proposed to solve this problem.Firstly,the affine linear model of the engine fuel rate is obtained based on engine test data.The piecewise linear approximation of the motor power demand is obtained considering the different energy flows in the charging and discharging stages of the battery.Then,the second-order Taylor expansion for the cost matrix at each time and state grid point is introduced to get the closed-form solution of the optimal torque split.The results show that this method can greatly reduce the computing burden by 93%while ensuring near-optimal fuel economy compared with the conventional DP method.

Research on System Control and Energy Management Strategy of Flux-Modulated Compound-Structure Permanent Magnet Synchronous Machine

The flux-modulated compound-structure permanent magnet synchronous machine (CS-PMSM), composed of a brushless double rotor machine (DRM) and a conventional permanent magnet synchronous machine (PMSM), is a power split device for plug-in hybrid electric vehicles. In this paper, its operating principle and mathematical model are introduced. A modified current controller with decoupled state feedback is proposed and verified. The system control strategy is simulated in Matlab, and the feasibility of the control system is proven. To improve fuel economy, an energy management strategy based on fuzzy logic controller is proposed and evaluated by the Urban Dynamometer Driving Schedule (UDDS) drive cycle. The results show that the total energy consumption is similar to that of Prius 2012.

基于A-ECMS的增程式电动汽车能量管理策略设计及应用

增程式电动汽车因其电驱效率高、无里程焦虑等优点,在新能源汽车市场备受青睐。为了进一步提升其综合性能,本研究在MATLAB/Simulink环境下构建了增程式电动汽车仿真模型,并利用等效自适应因子以改进燃油消耗最小化算法,形成了基于A-ECMS的能量管理策略。结果表明,在WLTC循环工况测试中,A-ECMS能量管理策略的百公里综合油耗为6.42 L,综合燃油消耗量为0.71 L,均低于其他方法。最终的电池充电状态(SOC)为31.1%,与目标SOC的偏差仅为1.1%。该策略显著提升了燃油经济性和电池使用寿命,为增程式电动汽车性能优化提供了新的参考方法。

Global Optimization‑Based Energy Management Strategy for Series–Parallel Hybrid Electric Vehicles Using Multi‑objective Optimization Algorithm

The study of series–parallel plug-in hybrid electric vehicles(PHEVs)has become a research hotspot in new energy vehicles.The global optimal Pareto solutions of energy management strategy(EMS)play a crucial role in the development of PHEVs.This paper presents a multi-objective global optimization algorithm for the EMS of PHEVs.The algorithm combines the Radau Pseudospectral Knotting Method(RPKM)and the Nondominated Sorting Genetic Algorithm(NSGA)-II to optimize both energy conservation and battery lifespan under the suburban driving conditions of the New European Driving Cycle.The driving conditions are divided into stages at evident mode switching points and the optimal objectives are computed using RPKM.The RPKM results serve as the fitness values in iteration through the NSGA-II method.The results of the algorithm applied to a PHEV simulation show a 26.74%–53.87%improvement in both objectives after 20 iterations compared to the solutions obtained using only RPKM.The proposed algorithm is evaluated against the weighting dynamic programming and is found to be close to the global optimality,with the added benefits of faster and more uniform solutions.

Online Learning Control for Hybrid Electric Vehicle

Improvements in hybrid electric vehicle (HEV) fuel economy and emissions heavily depend on an efficient energy management strategy (EMS). However, the uncertainty of future driving conditions generally cannot be easily tackled in EMS design. Most existing EMSs act upon fixed parameters and cannot adapt to varying driving conditions. Therefore, they usually fail to fully explore the potential of these advanced vehicles. In this paper, a novel EMS design procedure based on neural dynamic programming (NDP) is proposed. The NDP is a generic online learning algorithm, which combines stochastic dynamic programming (SDP) and the temporal difference (TD) method. Instead of computing the utility function and optimal control actions through Bellman equations, the NDP algorithm uses two neural networks to approximate them. The weights of these neural networks are updated online by the TD method. It avoids the high computational cost that SDP suffers from and is suitable for real-time implementation. The main advantages of NDP EMS is that it does not rely on prior information related to future driving conditions, and can self-tune with a wide variance in operating conditions. The NDP EMS has been applied to “Qianghua-I”, a prototype of a parallel HEV, using a revolving drum test bench for verification. Experiment results illustrate the potential of the proposed EMS in terms of fuel economy and in keeping state of charge (SOC) deviations at a low level. The proposed research ensures the optimality of NDP EMS, as well as real-time applicability.

Cloud Computing Based Optimal Driving for a Parallel Hybrid Electric Vehicle

A cloud computing based optimal driving method is proposed and its feasibility is validated through a real-world scenario simulation.Based on principles of vehicle dynamics,the driving optimization problem has been formulated into an optimal control problem constrained by traffic rules,directed at achieving lower equivalent fuel consumption and shorter travel time.In order to conveniently specify the constraints and facilitate the application of the dynamic programming(DP)algorithm,the driving optimization problem is transformed into spatial domain and discretized properly.Considering the heavy computational costs of the DP algorithm,a cloud computing based platform structure is proposed to solve the optimal driving problem in real-time.A case study is simulated based on a real-world traffic scenario in Matlab.Simulation results demonstrate that the cloud computing framework is promising toward realizing the real-time energy management for hybrid electric vehicles.

混动车辆能量管理模块化ECMS框架

文中提出一种用于混合动力车辆能量管理实时控制的模块化等效消耗最小化策略(ECMS)框架,可实现能量管理实时控制的分布式开发。该框架采用Pontryagin极小值原理求解ECMS最优控制问题。首先,将最优控制问题分解为与各个子系统相关的优化问题;其次,通过Hamiltonian函数将与每个子系统有关的优化问题协调到全局最优;最后,在某款DHT混合动力车辆验证模块化ECMS,此车辆模型支持4种操作模式:2种电动模式、并联模式和串联模式。测试结果表明,模块化ECMS只需修改能量管理系统中功率平衡方程的连接矩阵,即可解决上述每种操作模式的最优控制问题。此框架不需要迭代过程,更适合实时控制的分布式开发。