基于强化学习的多时隙铁路空车实时调配研究

Reinforcement-learning-based Multi-slot Rail Empty Wagon Real-time Distribution

铁路空车调配计划是进行运输组织的基础和重要条件,空车供求关系的时空变化特性和运输生产的动态性,使求解多时隙空车实时调配最优策略变得困难。强化学习中的Q-learning时序差分算法能较好地解决不完全信息下的大规模序列决策问题,故本文将决策周期划分为若干个时隙,提出多时隙空车实时调配模型:首先利用空车实际调配的局部马尔科夫特性改进Q-learning算法,进行“单一空车调配策略评估”以量化单一空车在决策周期内所有时空状态下采取不同行动的长期回报;然后提出空车实时优先调配算法,求解决策周期全局最优的调配策略。算例表明模型可以兼顾实时调配长期回报最大、空走距离小、即时需求响应程度高,求解出每时隙下最优且决策周期全局最优的实时调配策略,以使运输部门快速适应变化的货运市场需求、提供科学合理的空车实时调配策略是可行的。

Rail empty wagon distribution is critical to a transportation enterprise.The spatio-temporal characteristics of the supply and demand of empty wagons and the dynamics of transportation generate difficulties in developing an optimal strategy for multi-slot empty wagon real-time distribution.A Q-reinforcement-learning algorithm can solve large-scale sequence decision problems using incomplete information.In this study,the decision period is divided into multi-slots,and a multi-slot empty wagon distribution model is proposed.First,based on local Markov characteristics of empty wagon distribution,an improved Q-learning algorithm is designed,and a single empty wagon strategy evaluation is performed to evaluate a single wagon’s long-term gains under all spatio-temporal states during the decision period.Second,an empty wagon real-time priority distribution algorithm is proposed to solve the strategy for each slot.A case study of multi-slot empty wagon real-time distribution shows that our proposed model can maximize long-term gains as well as minimize unloaded distances of a real-time distribution.Thus,providing rail transportation enterprises with scientific real-time empty wagon distribution strategies is feasible.