通过优化地铁时刻表可有效降低地铁牵引能耗。为解决客流波动和车辆延误对实际节能率影响的问题,提出列车牵引和供电系统实时潮流计算分析模型和基于Dueling Deep Q Network(Dueling DQN)深度强化学习算法相结合的运行图节能优化方法,...通过优化地铁时刻表可有效降低地铁牵引能耗。为解决客流波动和车辆延误对实际节能率影响的问题,提出列车牵引和供电系统实时潮流计算分析模型和基于Dueling Deep Q Network(Dueling DQN)深度强化学习算法相结合的运行图节能优化方法,建立基于区间动态客流概率统计的时刻表迭代优化模型,降低动态客流变化对节能率的影响。对预测Q网络和目标Q网络分别选取自适应时刻估计和均方根反向传播方法,提高模型收敛快速性,同时以时刻表优化前、后总运行时间不变、乘客换乘时间和等待时间最小为优化目标,实现节能时刻表无感切换。以苏州轨道交通4号线为例验证方法的有效性,节能对比试验结果表明:在到达换乘站时刻偏差不超过2 s和列车全周转运行时间不变的前提下,列车牵引节能率达5.27%,车公里能耗下降4.99%。展开更多
With the rapid development ofmobile Internet,spatial crowdsourcing has becomemore andmore popular.Spatial crowdsourcing consists of many different types of applications,such as spatial crowd-sensing services.In terms ...With the rapid development ofmobile Internet,spatial crowdsourcing has becomemore andmore popular.Spatial crowdsourcing consists of many different types of applications,such as spatial crowd-sensing services.In terms of spatial crowd-sensing,it collects and analyzes traffic sensing data from clients like vehicles and traffic lights to construct intelligent traffic prediction models.Besides collecting sensing data,spatial crowdsourcing also includes spatial delivery services like DiDi and Uber.Appropriate task assignment and worker selection dominate the service quality for spatial crowdsourcing applications.Previous research conducted task assignments via traditional matching approaches or using simple network models.However,advanced mining methods are lacking to explore the relationship between workers,task publishers,and the spatio-temporal attributes in tasks.Therefore,in this paper,we propose a Deep Double Dueling Spatial-temporal Q Network(D3SQN)to adaptively learn the spatialtemporal relationship between task,task publishers,and workers in a dynamic environment to achieve optimal allocation.Specifically,D3SQNis revised through reinforcement learning by adding a spatial-temporal transformer that can estimate the expected state values and action advantages so as to improve the accuracy of task assignments.Extensive experiments are conducted over real data collected fromDiDi and ELM,and the simulation results verify the effectiveness of our proposed models.展开更多
Due to the high mileage and heavy load capabilities of hybrid electric vehicles(HEVs),energy management becomes crucial in improving energy efficiency.To avoid the over-dependence on the hard-crafted models,deep reinf...Due to the high mileage and heavy load capabilities of hybrid electric vehicles(HEVs),energy management becomes crucial in improving energy efficiency.To avoid the over-dependence on the hard-crafted models,deep reinforcement learning(DRL)is utilized to learn more precise energy management strategies(EMSs),but cannot generalize well to different driving situations in most cases.When driving cycles are changed,the neural network needs to be retrained,which is a time-consuming and laborious task.A more efficient transferable way is to combine DRL algorithms with transfer learning,which can utilize the knowledge of the driving cycles in other new driving situations,leading to better initial performance and a faster training process to convergence.In this paper,we propose a novel transferable EMS by incorporating the DRL method and dueling network architecture for HEVs.Simulation results indicate that the proposed method can generalize well to new driving cycles,with comparably initial performance and faster convergence in the training process.展开更多
针对认知无线电网络中多个次用户存在不同服务质量(quality of service,QoS)需求的频谱接入问题,提出了基于Dueling DQN(dueling deep Q-network)的分布式动态频谱接入方法。该方法通过与环境交互学习实现在次用户不掌握系统信道先验信...针对认知无线电网络中多个次用户存在不同服务质量(quality of service,QoS)需求的频谱接入问题,提出了基于Dueling DQN(dueling deep Q-network)的分布式动态频谱接入方法。该方法通过与环境交互学习实现在次用户不掌握系统信道先验信息条件下动态获得最佳频谱接入策略,并以次用户碰撞次数以及成功接入信道次数分析比较所提出方法的性能。仿真结果表明,提出的方法在保护主用户不受干扰、满足多异质用户QoS需求的前提下,能够有效减少次用户间碰撞次数,提高次用户成功接入信道次数,相比随机接入与短视策略(myopic policy)频谱接入方法,该方法的碰撞次数分别降低60%和90%,其成功接入性能分别提高30%和50%。展开更多
Unmanned aerial vehicles(UAVs) are increasingly considered in safe autonomous navigation systems to explore unknown environments where UAVs are equipped with multiple sensors to perceive the surroundings. However, how...Unmanned aerial vehicles(UAVs) are increasingly considered in safe autonomous navigation systems to explore unknown environments where UAVs are equipped with multiple sensors to perceive the surroundings. However, how to achieve UAVenabled data dissemination and also ensure safe navigation synchronously is a new challenge. In this paper, our goal is minimizing the whole weighted sum of the UAV’s task completion time while satisfying the data transmission task requirement and the UAV’s feasible flight region constraints. However, it is unable to be solved via standard optimization methods mainly on account of lacking a tractable and accurate system model in practice. To overcome this tough issue,we propose a new solution approach by utilizing the most advanced dueling double deep Q network(dueling DDQN) with multi-step learning. Specifically, to improve the algorithm, the extra labels are added to the primitive states. Simulation results indicate the validity and performance superiority of the proposed algorithm under different data thresholds compared with two other benchmarks.展开更多
Equipment development planning(EDP)is usually a long-term process often performed in an environment with high uncertainty.The traditional multi-stage dynamic programming cannot cope with this kind of uncertainty with ...Equipment development planning(EDP)is usually a long-term process often performed in an environment with high uncertainty.The traditional multi-stage dynamic programming cannot cope with this kind of uncertainty with unpredictable situations.To deal with this problem,a multi-stage EDP model based on a deep reinforcement learning(DRL)algorithm is proposed to respond quickly to any environmental changes within a reasonable range.Firstly,the basic problem of multi-stage EDP is described,and a mathematical planning model is constructed.Then,for two kinds of uncertainties(future capabi lity requirements and the amount of investment in each stage),a corresponding DRL framework is designed to define the environment,state,action,and reward function for multi-stage EDP.After that,the dueling deep Q-network(Dueling DQN)algorithm is used to solve the multi-stage EDP to generate an approximately optimal multi-stage equipment development scheme.Finally,a case of ten kinds of equipment in 100 possible environments,which are randomly generated,is used to test the feasibility and effectiveness of the proposed models.The results show that the algorithm can respond instantaneously in any state of the multistage EDP environment and unlike traditional algorithms,the algorithm does not need to re-optimize the problem for any change in the environment.In addition,the algorithm can flexibly adjust at subsequent planning stages in the event of a change to the equipment capability requirements to adapt to the new requirements.展开更多
基金supported in part by the Pioneer and Leading Goose R&D Program of Zhejiang Province under Grant 2022C01083 (Dr.Yu Li,https://zjnsf.kjt.zj.gov.cn/)Pioneer and Leading Goose R&D Program of Zhejiang Province under Grant 2023C01217 (Dr.Yu Li,https://zjnsf.kjt.zj.gov.cn/).
文摘With the rapid development ofmobile Internet,spatial crowdsourcing has becomemore andmore popular.Spatial crowdsourcing consists of many different types of applications,such as spatial crowd-sensing services.In terms of spatial crowd-sensing,it collects and analyzes traffic sensing data from clients like vehicles and traffic lights to construct intelligent traffic prediction models.Besides collecting sensing data,spatial crowdsourcing also includes spatial delivery services like DiDi and Uber.Appropriate task assignment and worker selection dominate the service quality for spatial crowdsourcing applications.Previous research conducted task assignments via traditional matching approaches or using simple network models.However,advanced mining methods are lacking to explore the relationship between workers,task publishers,and the spatio-temporal attributes in tasks.Therefore,in this paper,we propose a Deep Double Dueling Spatial-temporal Q Network(D3SQN)to adaptively learn the spatialtemporal relationship between task,task publishers,and workers in a dynamic environment to achieve optimal allocation.Specifically,D3SQNis revised through reinforcement learning by adding a spatial-temporal transformer that can estimate the expected state values and action advantages so as to improve the accuracy of task assignments.Extensive experiments are conducted over real data collected fromDiDi and ELM,and the simulation results verify the effectiveness of our proposed models.
文摘Due to the high mileage and heavy load capabilities of hybrid electric vehicles(HEVs),energy management becomes crucial in improving energy efficiency.To avoid the over-dependence on the hard-crafted models,deep reinforcement learning(DRL)is utilized to learn more precise energy management strategies(EMSs),but cannot generalize well to different driving situations in most cases.When driving cycles are changed,the neural network needs to be retrained,which is a time-consuming and laborious task.A more efficient transferable way is to combine DRL algorithms with transfer learning,which can utilize the knowledge of the driving cycles in other new driving situations,leading to better initial performance and a faster training process to convergence.In this paper,we propose a novel transferable EMS by incorporating the DRL method and dueling network architecture for HEVs.Simulation results indicate that the proposed method can generalize well to new driving cycles,with comparably initial performance and faster convergence in the training process.
文摘针对认知无线电网络中多个次用户存在不同服务质量(quality of service,QoS)需求的频谱接入问题,提出了基于Dueling DQN(dueling deep Q-network)的分布式动态频谱接入方法。该方法通过与环境交互学习实现在次用户不掌握系统信道先验信息条件下动态获得最佳频谱接入策略,并以次用户碰撞次数以及成功接入信道次数分析比较所提出方法的性能。仿真结果表明,提出的方法在保护主用户不受干扰、满足多异质用户QoS需求的前提下,能够有效减少次用户间碰撞次数,提高次用户成功接入信道次数,相比随机接入与短视策略(myopic policy)频谱接入方法,该方法的碰撞次数分别降低60%和90%,其成功接入性能分别提高30%和50%。
基金supported by the National Natural Science Foundation of China (No. 61931011)。
文摘Unmanned aerial vehicles(UAVs) are increasingly considered in safe autonomous navigation systems to explore unknown environments where UAVs are equipped with multiple sensors to perceive the surroundings. However, how to achieve UAVenabled data dissemination and also ensure safe navigation synchronously is a new challenge. In this paper, our goal is minimizing the whole weighted sum of the UAV’s task completion time while satisfying the data transmission task requirement and the UAV’s feasible flight region constraints. However, it is unable to be solved via standard optimization methods mainly on account of lacking a tractable and accurate system model in practice. To overcome this tough issue,we propose a new solution approach by utilizing the most advanced dueling double deep Q network(dueling DDQN) with multi-step learning. Specifically, to improve the algorithm, the extra labels are added to the primitive states. Simulation results indicate the validity and performance superiority of the proposed algorithm under different data thresholds compared with two other benchmarks.
基金supported by the National Natural Science Foundation of China(71690233,72001209)the Scientific Research Foundation of the National University of Defense Technology(ZK19-16)。
文摘Equipment development planning(EDP)is usually a long-term process often performed in an environment with high uncertainty.The traditional multi-stage dynamic programming cannot cope with this kind of uncertainty with unpredictable situations.To deal with this problem,a multi-stage EDP model based on a deep reinforcement learning(DRL)algorithm is proposed to respond quickly to any environmental changes within a reasonable range.Firstly,the basic problem of multi-stage EDP is described,and a mathematical planning model is constructed.Then,for two kinds of uncertainties(future capabi lity requirements and the amount of investment in each stage),a corresponding DRL framework is designed to define the environment,state,action,and reward function for multi-stage EDP.After that,the dueling deep Q-network(Dueling DQN)algorithm is used to solve the multi-stage EDP to generate an approximately optimal multi-stage equipment development scheme.Finally,a case of ten kinds of equipment in 100 possible environments,which are randomly generated,is used to test the feasibility and effectiveness of the proposed models.The results show that the algorithm can respond instantaneously in any state of the multistage EDP environment and unlike traditional algorithms,the algorithm does not need to re-optimize the problem for any change in the environment.In addition,the algorithm can flexibly adjust at subsequent planning stages in the event of a change to the equipment capability requirements to adapt to the new requirements.
