基于深度强化学习的网联混合动力汽车队列控制

Deep Reinforcement Learning-based Control Strategy of Connected Hybrid Electric Vehicles Platooning

针对网联混合动力汽车队列具有混杂非线性、多动力源混合驱动等特点,提出基于深度强化学习的队列分层控制策略。首先,设计了队列模型预测控制器,基于车车通信获取的车辆状态求解约束条件下满足多性能目标的队列车辆最优期望加速度。其次,为提高车辆的燃油经济性,将发动机最优工作曲线和电池特性曲线作为专家知识嵌入深度强化学习算法中。然后,通过分析电池荷电状态、车辆车速以及车辆加速度对智能体动作值的影响来阐明基于深度强化学习(Deep Q network,DQN)的队列能量管理策略是如何根据动作值实现对队列中车辆多系统动力输出之间的协调控制。最后,设计了以电池荷电状态、瞬时燃油消耗率为自变量的奖励值函数,利用最小化损失函数,采用梯度下降法对DQN网络参数进行更新,通过深度强化学习算法实现网联混合动力汽车队列的能量管理控制。试验结果表明,所提出的队列控制策略可以动态规划出队列中车辆期望加速度,实时合理的分配发动机功率与电机功率,最终实现队列中车辆的节能行驶。

In view of the characteristics of strong nonlinearity and hybrid driving with multi-power sources,a deep learning based hierarchical control strategy of connected hybrid electric vehicles platooning is proposed.Firstly,a model predictive controller for platoon is designed to solve the expected optimal and multi-objective acceleration of vehicles based on the vehicular state acquiring by vehicle to vehicle communication.Secondly,the optimal operating curve of the engine and the battery characteristic curve are embedded into the deep learning algorithm as the expert knowledge.Then,the influence of battery power,engine power,vehicle speeds and vehicle accelerations on the action value of the agent is discussed to illustrate how the DQN deep learning-based energy management control algorithm realizes the coordinated control of the multi-system power output of vehicles in the platoon according to the action value.Finally,a reward function with battery state of charge and instantaneous fuel consumption as independent variables is designed.By using the minimization loss function,the parameters of the DQN network are updated by the gradient descending method,and the energy management control is realized by the deep reinforcement learning method.The test results show that the proposed control strategy can dynamically plan the expected acceleration of vehicles in the platoon,and reasonably allocate between the engine power and motor power in real time,and finally achieve energy-saving driving of vehicles in the platoon.