Transformer-Aided Deep Double Dueling Spatial-Temporal Q-Network for Spatial Crowdsourcing Analysis

With the rapid development ofmobile Internet,spatial crowdsourcing has becomemore andmore popular.Spatial crowdsourcing consists of many different types of applications,such as spatial crowd-sensing services.In terms of spatial crowd-sensing,it collects and analyzes traffic sensing data from clients like vehicles and traffic lights to construct intelligent traffic prediction models.Besides collecting sensing data,spatial crowdsourcing also includes spatial delivery services like DiDi and Uber.Appropriate task assignment and worker selection dominate the service quality for spatial crowdsourcing applications.Previous research conducted task assignments via traditional matching approaches or using simple network models.However,advanced mining methods are lacking to explore the relationship between workers,task publishers,and the spatio-temporal attributes in tasks.Therefore,in this paper,we propose a Deep Double Dueling Spatial-temporal Q Network(D3SQN)to adaptively learn the spatialtemporal relationship between task,task publishers,and workers in a dynamic environment to achieve optimal allocation.Specifically,D3SQNis revised through reinforcement learning by adding a spatial-temporal transformer that can estimate the expected state values and action advantages so as to improve the accuracy of task assignments.Extensive experiments are conducted over real data collected fromDiDi and ELM,and the simulation results verify the effectiveness of our proposed models.

Automatic depth matching method of well log based on deep reinforcement learning

In the traditional well log depth matching tasks,manual adjustments are required,which means significantly labor-intensive for multiple wells,leading to low work efficiency.This paper introduces a multi-agent deep reinforcement learning(MARL)method to automate the depth matching of multi-well logs.This method defines multiple top-down dual sliding windows based on the convolutional neural network(CNN)to extract and capture similar feature sequences on well logs,and it establishes an interaction mechanism between agents and the environment to control the depth matching process.Specifically,the agent selects an action to translate or scale the feature sequence based on the double deep Q-network(DDQN).Through the feedback of the reward signal,it evaluates the effectiveness of each action,aiming to obtain the optimal strategy and improve the accuracy of the matching task.Our experiments show that MARL can automatically perform depth matches for well-logs in multiple wells,and reduce manual intervention.In the application to the oil field,a comparative analysis of dynamic time warping(DTW),deep Q-learning network(DQN),and DDQN methods revealed that the DDQN algorithm,with its dual-network evaluation mechanism,significantly improves performance by identifying and aligning more details in the well log feature sequences,thus achieving higher depth matching accuracy.

Associative Tasks Computing Offloading Scheme in Internet of Medical Things with Deep Reinforcement Learning

The Internet of Medical Things(Io MT) is regarded as a critical technology for intelligent healthcare in the foreseeable 6G era. Nevertheless, due to the limited computing power capability of edge devices and task-related coupling relationships, Io MT faces unprecedented challenges. Considering the associative connections among tasks, this paper proposes a computing offloading policy for multiple-user devices(UDs) considering device-to-device(D2D) communication and a multi-access edge computing(MEC)technique under the scenario of Io MT. Specifically,to minimize the total delay and energy consumption concerning the requirement of Io MT, we first analyze and model the detailed local execution, MEC execution, D2D execution, and associated tasks offloading exchange model. Consequently, the associated tasks’ offloading scheme of multi-UDs is formulated as a mixed-integer nonconvex optimization problem. Considering the advantages of deep reinforcement learning(DRL) in processing tasks related to coupling relationships, a Double DQN based associative tasks computing offloading(DDATO) algorithm is then proposed to obtain the optimal solution, which can make the best offloading decision under the condition that tasks of UDs are associative. Furthermore, to reduce the complexity of the DDATO algorithm, the cacheaided procedure is intentionally introduced before the data training process. This avoids redundant offloading and computing procedures concerning tasks that previously have already been cached by other UDs. In addition, we use a dynamic ε-greedy strategy in the action selection section of the algorithm, thus preventing the algorithm from falling into a locally optimal solution. Simulation results demonstrate that compared with other existing methods for associative task models concerning different structures in the Io MT network, the proposed algorithm can lower the total cost more effectively and efficiently while also providing a tradeoff between delay and energy consumption tolerance.

基于引导Minimax-DDQN的无人机空战机动决策

针对无人机(UAV)空战环境信息复杂、对抗性强所导致的敌机机动策略难以预测,以及作战胜率不高的问题,设计了一种引导Minimax-DDQN(Minimax-Double Deep Q-Network)算法。首先,在Minimax决策方法的基础上提出了一种引导式策略探索机制;然后,结合引导Minimax策略,以提升Q网络更新效率为出发点设计了一种DDQN(Double Deep Q-Network)算法;最后,提出进阶式三阶段的网络训练方法,通过不同决策模型间的对抗训练,获取更为优化的决策模型。实验结果表明,相较于Minimax-DQN(Minimax-DQN)、Minimax-DDQN等算法,所提算法追击直线目标的成功率提升了14%~60%,并且与DDQN算法的对抗胜率不低于60%。可见,与DDQN、Minimax-DDQN等算法相比,所提算法在高对抗的作战环境中具有更强的决策能力,适应性更好。

未知环境下改进DDQN的无人机探索航迹规划研究

对未知环境的探索,如搜救、追逃等场景,无人机需要一边探索(感知)环境一边完成当前的航迹规划(动作选择)。针对上述场景,为了提高无人机对未知环境的探索范围,提出了结合长短期记忆的改进深度双Q网络探索航迹规划方法:搭建仿真地图,以无人机视野内的环境信息作为输入,引入长短期记忆网络,输出动作方向的选择;设置探索经验样本优先级,提高训练效率;加入飞行动力学约束,设计合理的状态、动作空间及单步奖励函数。运用所提算法,无人机可以自主规划出一条无碰撞且对环境探索范围大的航迹。仿真实验结果表明:在未知环境下,所提算法得到的探索面积比、单步探索平均奖励值等指标均优于传统的DDQN算法。

Intelligent Voltage Control Method in Active Distribution Networks Based on Averaged Weighted Double Deep Q-network Algorithm

High penetration of distributed renewable energy sources and electric vehicles(EVs)makes future active distribution network(ADN)highly variable.These characteristics put great challenges to traditional voltage control methods.Voltage control based on the deep Q-network(DQN)algorithm offers a potential solution to this problem because it possesses humanlevel control performance.However,the traditional DQN methods may produce overestimation of action reward values,resulting in degradation of obtained solutions.In this paper,an intelligent voltage control method based on averaged weighted double deep Q-network(AWDDQN)algorithm is proposed to overcome the shortcomings of overestimation of action reward values in DQN algorithm and underestimation of action reward values in double deep Q-network(DDQN)algorithm.Using the proposed method,the voltage control objective is incorporated into the designed action reward values and normalized to form a Markov decision process(MDP)model which is solved by the AWDDQN algorithm.The designed AWDDQN-based intelligent voltage control agent is trained offline and used as online intelligent dynamic voltage regulator for the ADN.The proposed voltage control method is validated using the IEEE 33-bus and 123-bus systems containing renewable energy sources and EVs,and compared with the DQN and DDQN algorithms based methods,and traditional mixed-integer nonlinear program based methods.The simulation results show that the proposed method has better convergence and less voltage volatility than the other ones.

Deep reinforcement learning for UAV swarm rendezvous behavior

The unmanned aerial vehicle(UAV)swarm technology is one of the research hotspots in recent years.With the continuous improvement of autonomous intelligence of UAV,the swarm technology of UAV will become one of the main trends of UAV development in the future.This paper studies the behavior decision-making process of UAV swarm rendezvous task based on the double deep Q network(DDQN)algorithm.We design a guided reward function to effectively solve the problem of algorithm convergence caused by the sparse return problem in deep reinforcement learning(DRL)for the long period task.We also propose the concept of temporary storage area,optimizing the memory playback unit of the traditional DDQN algorithm,improving the convergence speed of the algorithm,and speeding up the training process of the algorithm.Different from traditional task environment,this paper establishes a continuous state-space task environment model to improve the authentication process of UAV task environment.Based on the DDQN algorithm,the collaborative tasks of UAV swarm in different task scenarios are trained.The experimental results validate that the DDQN algorithm is efficient in terms of training UAV swarm to complete the given collaborative tasks while meeting the requirements of UAV swarm for centralization and autonomy,and improving the intelligence of UAV swarm collaborative task execution.The simulation results show that after training,the proposed UAV swarm can carry out the rendezvous task well,and the success rate of the mission reaches 90%.

基于双深度Q网络算法的多用户端对端能源共享机制研究

端对端(P2P)电力交易作为用户侧能源市场的一种新的能源平衡和互动方式,可以有效促进用户群体内的能源共享,提高参与能源市场用户的经济效益。然而传统求解用户间P2P交易的方法依赖对于光伏、负荷数据的预测,难以实时响应用户间的源荷变动问题。为此,本文建立了一种以多类型用户为基础的多用户P2P能源社区交易模型,并引入基于双深度Q网络(DDQN)的强化学习(RL)算法对其进行求解。所提方法通过DDQN算法中的预测网络以及目标网络读取多用户P2P能源社区中的环境信息,训练后的神经网络可通过实时的光伏、负荷以及电价数据对当前社区内的多用户P2P交易问题进行求解。案例仿真结果表明,所提方法在促进社区内用户间P2P能源交易共享的同时,保证了多用户P2P能源社区的经济性。

货运铁路会让列车调度指挥与运行控制协同优化研究

针对既有货运铁路调度集中系统和列车运行控制系统分立设置,列车会让质量高度依赖调度员和司机的经验水平,突发事件响应慢等问题,构建货运铁路协同优化仿真系统,实现列车会让过程中调度指挥与运行控制的有机统一。首先依据货运铁路列车会让过程,增加通过列车和对向列车的位置、速度,以及列车牵引制动特性、线路条件等约束,构建列车协同会让优化模型;然后采用双深度Q网络(DDQN)进行模型求解,在计算列车速度曲线和选择列车工况序列时,考虑对向列车速度和距会让点的距离,协同优化通过、会让列车的运行过程。最后以某地方铁路实际线路、列车数据为例,验证模型和算法的可行性和适应性。试验结果表明,该系统可辅助调度人员实时掌握列车运行状态,有效调整列车运行速度,提高列车会让运输效率。

基于DDQN的片上网络混合关键性消息调度方法

对片上网络(NoC)承载的混合关键性消息进行实时调度是其应用于航空电子系统片上多核通信的关键。为解决可满足性模理论(SMT)法求解效率低、低优先级消息等待延迟大的问题,提出了一种基于双深度Q网络(DDQN)的混合关键性消息调度方法。将虫孔交换机制下的消息调度问题建模为马尔可夫决策过程,建立包含环境、动作、状态、奖励的多层感知调度模型;随机生成多组分布不同的混合关键性消息作为训练样本,采用DDQN算法求解该调度模型;在此基础上,提出并实现了带孔隙DDQN算法,在保证时间触发(TT)消息可调度前提下为速率约束(RC)消息预留用于虫孔交换的时隙。算例研究表明:所提方法的求解时长及TT消息确定性端到端延迟的平均值均低于SMT法;带孔隙DDQN算法的RC消息延迟较不带孔隙DDQN算法和SMT法显著降低。

Autonomous maneuver decision-making for a UCAV in short-range aerial combat based on an MS-DDQN algorithm

To solve the problem of realizing autonomous aerial combat decision-making for unmanned combat aerial vehicles(UCAVs) rapidly and accurately in an uncertain environment, this paper proposes a decision-making method based on an improved deep reinforcement learning(DRL) algorithm: the multistep double deep Q-network(MS-DDQN) algorithm. First, a six-degree-of-freedom UCAV model based on an aircraft control system is established on a simulation platform, and the situation assessment functions of the UCAV and its target are established by considering their angles, altitudes, environments, missile attack performances, and UCAV performance. By controlling the flight path angle, roll angle, and flight velocity, 27 common basic actions are designed. On this basis, aiming to overcome the defects of traditional DRL in terms of training speed and convergence speed, the improved MS-DDQN method is introduced to incorporate the final return value into the previous steps. Finally, the pre-training learning model is used as the starting point for the second learning model to simulate the UCAV aerial combat decision-making process based on the basic training method, which helps to shorten the training time and improve the learning efficiency. The improved DRL algorithm significantly accelerates the training speed and estimates the target value more accurately during training, and it can be applied to aerial combat decision-making.

Safe Navigation for UAV-Enabled Data Dissemination by Deep Reinforcement Learning in Unknown Environments

Unmanned aerial vehicles(UAVs) are increasingly considered in safe autonomous navigation systems to explore unknown environments where UAVs are equipped with multiple sensors to perceive the surroundings. However, how to achieve UAVenabled data dissemination and also ensure safe navigation synchronously is a new challenge. In this paper, our goal is minimizing the whole weighted sum of the UAV’s task completion time while satisfying the data transmission task requirement and the UAV’s feasible flight region constraints. However, it is unable to be solved via standard optimization methods mainly on account of lacking a tractable and accurate system model in practice. To overcome this tough issue,we propose a new solution approach by utilizing the most advanced dueling double deep Q network(dueling DDQN) with multi-step learning. Specifically, to improve the algorithm, the extra labels are added to the primitive states. Simulation results indicate the validity and performance superiority of the proposed algorithm under different data thresholds compared with two other benchmarks.

基于动作注意策略的树形DDQN目标候选区域提取方法

针对机器人在家庭环境下的目标检测问题,该文提出一种基于动作注意策略的树形双深度Q网络(TDDQN)目标候选区域提取的方法,该方法将双深度Q网络(DDQN)的方法与树结构的方法相结合,通过执行改变检测框的动作以使目标逐渐集中在检测框内。首先采用DDQN方法在执行较少的动作后选择出当前状态的最佳动作,获取符合条件的候选区域。然后根据执行所选择动作之后所得到的状态重复执行上述过程,以此构成树结构的多条"最佳"路径。最后采用非极大值抑制的方法从多个符合条件的候选区域选择出最佳候选区域。在PascalVOC2007以及Pascal VOC2012上的实验结果表明,在不同数量的候选区域、不同阈值的IoU和不同大小以及不同种类对象的实验条件下,所提方法较其他方法都有着更好的检测性能,可以较好地实现目标检测。

基于深度强化学习的Boost变换器控制策略

针对Boost变换器母线电压稳定控制存在模型不确定性和非线性的问题,提出了一种基于无模型深度强化学习的智能控制策略。结合强化学习DDQN(double DQN)算法与DDPG算法设计了Boost变换器控制器,包括了状态、动作空间、奖励函数以及神经网络的设计以提高控制器动态性能;基于ModelicaGym库开发工具包reinforment learning modelica(RLM)实现了Boost变换器模型与强化学习智能体的联合仿真。通过与双环PI控制器的对比仿真表明:强化学习控制器在三种工况下的母线电压稳定控制结果具有更好的动态性能。

基于深度强化学习的平滑车速控制系统

为调整不同路段的限速值,平滑交通流,从而提升高速公路车辆通行的安全性和效率,针对交通瓶颈区设计一种基于深度强化学习的平滑车速管控系统。该系统主要包含动态限速启动、限速值确定与更新和情报板动态发布等3个模块。将深度强化学习算法DDQN(Double Deep Q-Network)引入系统中,提出一种基于DDQN的平滑车速控制策略,从目标网络和经验回顾2个维度提升该算法的性能。基于元胞传输模型(Cellular Transmission Model,CTM)对宁夏高速公路某路段的交通流运行场景进行仿真,以车辆总通行时间和车流量为评价指标验证该系统的有效性,结果表明该系统能提高瓶颈区内拥堵路段车辆的通行效率。

基于深度强化学习的移动机器人路径规划

为解决传统的深度Q网络模型下机器人探索复杂未知环境时收敛速度慢的问题,提出了基于竞争网络结构的改进深度双Q网络方法(Improved Dueling Deep Double Q-Network,IDDDQN)。移动机器人通过改进的DDQN网络结构对其三个动作的值函数进行估计,并更新网络参数,通过训练网络得到相应的Q值。移动机器人采用玻尔兹曼分布与ε-greedy相结合的探索策略,选择一个最优动作,到达下一个观察。机器人将通过学习收集到的数据采用改进的重采样优选机制存储到缓存记忆单元中,并利用小批量数据训练网络。实验结果显示,与基本DDQN算法比,IDDDQN训练的机器人能够更快地适应未知环境,网络的收敛速度也得到提高,到达目标点的成功率增加了3倍多,在未知的复杂环境中可以更好地获取最优路径。

基于深度强化学习的智能灯个性化调节方法

提出一种基于深度强化学习的智能灯亮度个性化调节方法,综合考虑自然光亮度及用户位置对用户实际感受亮度的影响,动态计算并设置灯光亮度,以满足用户个性化使用习惯。在每次完成灯光亮度自动调节后,根据用户是否再次进行手动调节设定正、负反馈,训练强化学习模型逐渐拟合用户使用习惯。实验分别实现了DQN、DDQN和A3C三种算法,在基于DIALux环境产生的数据集上进行对比分析,并给出原型系统的软硬件实现。

基于深度强化学习的股市操盘手模型研究

股票市场具有变化快、干扰因素多、周期数据不足等特点,股票交易是一种不完全信息下的博弈过程,单目标的监督学习模型很难处理这类序列化决策问题。强化学习是解决该类问题的有效途径之一。提出了基于深度强化学习的智能股市操盘手模型ISTG(Intelligent Stock Trader and Gym),融合历史行情数据、技术指标、宏观经济指标等多数据类型,分析评判标准和优秀控制策略,加工长周期数据,实现可增量扩展不同类型数据的复盘模型,自动计算回报标签,训练智能操盘手,并提出直接利用行情数据计算单步确定性动作值的方法。采用中国股市1400多支的有10年以上数据的股票进行多种对比实验,ISTG的总体收益达到13%,优于买入持有总体−7%的表现。

基于强化学习的六足机器人动态避障研究

针对局部可观测的非线性动态地震环境下,六足机器人采用传统算法进行动态避障时易出现算法不稳定的情况。运用了基于双重深度Q网络(DDQN)的决策方式,通过传感器数据输入卷积神经网络(CNN)并结合强化学习的策略,下达命令到六足机器人,控制输出决策动作,实现机器人动态避障。将系统的环境反馈与决策控制直接形成闭环,通过最大化机器人与避障环境交互产生的累计奖励回报,更新神经网络权重,形成最优决策策略。通过六足机器人平台实验结果证明:此方法能较好地减少传统深度强化学习算法容易导致过度估计状态动作值和损失函数难以收敛的风险;并且提高了六足机器人进行动态避障的效率和稳定性。

优先状态估计的双深度Q网络

深度强化学习探索问题中,需要根据环境给予的外部奖赏以作出决策,而在稀疏奖赏环境下,训练初期将获取不到任何信息,且在训练后期难以动态地结合已获得的信息对探索策略进行调整。为缓解这个问题,提出优先状态估计方法,在对状态进行访问时给予优先值,结合外部奖赏一并存入经验池中,引导探索的策略方向。结合DDQN(Double Deep Q Network)与优先经验回放,在OpenAI Gym中的MountainCar经典控制问题与Atari 2600中的FreeWay游戏中进行对比实验,结果表明该方法在稀疏奖赏环境中具有更好的学习性能,取得了更高的平均分数。