基于图像卷积神经网络的匝道控制深度强化学习算法研究

Research on Ramp Metering Based on Deep Reinforcement Learning with Image Convolutional Neural Network

针对入口匝道流量控制提升通行效率问题,提出了一种基于图像卷积神经网络的匝道控制深度强化学习算法.克服传统依赖定点检测器的匝道控制对于交通状态估计模糊的缺陷,建立基于图像卷积神经网络的连续时空交通状态解析,采用具有优先经验回放的深度Q学习算法,构建以视频图像作为输入、最优匝道流量策略为输出的算法框架.基于交通仿真(SUMO)平台,模拟了典型的高速公路合流瓶颈路段并进行控制效果测试.结果表明,深度强化学习匝道控制策略能够主动响应不同的交通状态,在短训练时间内达到目标找到最优控制策略,通过采取合适控制动作消除和预防合流区拥堵.本文提出的控制策略有效减少系统总旅行时间13.05%,优于传统定时调节式匝道控制和反馈式匝道控制算法,能更加有效提升高速公路合流区通行效率.

To improve the efficiency of the roadway traffic system, this study proposes a ramp metering strategy based on deep reinforcement learning with image convolutional neural networks. This strategy overcomes the blurred state estimation from traditional fixed loop detectors in the previous ramp metering strategy, and performs a continuous space-time traffic state analysis on the basis of the image convolutional neural networks. The deep Q network (DQN) algorithm with a prioritized experience replay is used to construct a methodological framework which takes the image as inputs and generates the optimal metering. The open-source software SUMO, a simulation platform, is employed to model the highway weaving bottleneck and evaluate the ramp metering strategy. The results show that the deep reinforcement learning ramp metering strategy can actively respond to different traffic states, converge to find the optimal control strategy in short training time and take appropriate actions to alleviate and prevent congestion. The strategy proposed in this paper effectively reduces the total system travel time by 13.05%, which effectively improves the traffic efficiency of the highway bottleneck and is superior to the traditional fixed-time ramp metering and reactive ramp metering strategy.