Transformer-Aided Deep Double Dueling Spatial-Temporal Q-Network for Spatial Crowdsourcing Analysis

With the rapid development ofmobile Internet,spatial crowdsourcing has becomemore andmore popular.Spatial crowdsourcing consists of many different types of applications,such as spatial crowd-sensing services.In terms of spatial crowd-sensing,it collects and analyzes traffic sensing data from clients like vehicles and traffic lights to construct intelligent traffic prediction models.Besides collecting sensing data,spatial crowdsourcing also includes spatial delivery services like DiDi and Uber.Appropriate task assignment and worker selection dominate the service quality for spatial crowdsourcing applications.Previous research conducted task assignments via traditional matching approaches or using simple network models.However,advanced mining methods are lacking to explore the relationship between workers,task publishers,and the spatio-temporal attributes in tasks.Therefore,in this paper,we propose a Deep Double Dueling Spatial-temporal Q Network(D3SQN)to adaptively learn the spatialtemporal relationship between task,task publishers,and workers in a dynamic environment to achieve optimal allocation.Specifically,D3SQNis revised through reinforcement learning by adding a spatial-temporal transformer that can estimate the expected state values and action advantages so as to improve the accuracy of task assignments.Extensive experiments are conducted over real data collected fromDiDi and ELM,and the simulation results verify the effectiveness of our proposed models.

基于DQN和功率分配的FDA-MIMO雷达抗扫频干扰

频率分集阵列(Frequency Diversity Array,FDA)雷达由于其阵列元件的频率增量产生了许多新的特性,包括其可以通过发射功率分配进行灵活的发射波形频谱控制。在以扫频干扰为电磁干扰环境的假设下,首先,通过引入强化学习的框架,建立了频率分集阵列-多输入多输出(Frequency Diversity Array-Multiple Input Multiple Output,FDA-MIMO)雷达与电磁干扰环境交互模型,使得FDA-MIMO雷达能够在与电磁环境交互过程中,感知干扰抑制干扰。其次,本文提出了一种基于深度Q网络(Deep Q-Network,DQN)和FDA-MIMO雷达发射功率分配的扫频干扰抑制方法,使得雷达系统能够在充分利用频谱资源的情况下最大化SINR。最后,仿真结果证实,在强化学习框架下,FDA-MIMO雷达能够通过对发射功率分配进行优化,完成干扰抑制,提升雷达性能。

Reinforcement Learning with an Ensemble of Binary Action Deep Q-Networks

With the advent of Reinforcement Learning(RL)and its continuous progress,state-of-the-art RL systems have come up for many challenging and real-world tasks.Given the scope of this area,various techniques are found in the literature.One such notable technique,Multiple Deep Q-Network(DQN)based RL systems use multiple DQN-based-entities,which learn together and communicate with each other.The learning has to be distributed wisely among all entities in such a scheme and the inter-entity communication protocol has to be carefully designed.As more complex DQNs come to the fore,the overall complexity of these multi-entity systems has increased many folds leading to issues like difficulty in training,need for high resources,more training time,and difficulty in fine-tuning leading to performance issues.Taking a cue from the parallel processing found in the nature and its efficacy,we propose a lightweight ensemble based approach for solving the core RL tasks.It uses multiple binary action DQNs having shared state and reward.The benefits of the proposed approach are overall simplicity,faster convergence and better performance compared to conventional DQN based approaches.The approach can potentially be extended to any type of DQN by forming its ensemble.Conducting extensive experimentation,promising results are obtained using the proposed ensemble approach on OpenAI Gym tasks,and Atari 2600 games as compared to recent techniques.The proposed approach gives a stateof-the-art score of 500 on the Cartpole-v1 task,259.2 on the LunarLander-v2 task,and state-of-the-art results on four out of five Atari 2600 games.

UAV Autonomous Navigation for Wireless Powered Data Collection with Onboard Deep Q-Network

In a rechargeable wireless sensor network,utilizing the unmanned aerial vehicle(UAV)as a mobile base station(BS)to charge sensors and collect data effectively prolongs the network’s lifetime.In this paper,we jointly optimize the UAV’s flight trajectory and the sensor selection and operation modes to maximize the average data traffic of all sensors within a wireless sensor network(WSN)during finite UAV’s flight time,while ensuring the energy required for each sensor by wireless power transfer(WPT).We consider a practical scenario,where the UAV has no prior knowledge of sensor locations.The UAV performs autonomous navigation based on the status information obtained within the coverage area,which is modeled as a Markov decision process(MDP).The deep Q-network(DQN)is employed to execute the navigation based on the UAV position,the battery level state,channel conditions and current data traffic of sensors within the UAV’s coverage area.Our simulation results demonstrate that the DQN algorithm significantly improves the network performance in terms of the average data traffic and trajectory design.

未知环境下基于Dueling DQN的无人机路径规划研究

为有效解决无人机在未知环境下的路径规划问题,提出一种基于Dueling DQN的路径规划方法。首先,在DQN的基础上,引入对抗网络架构,从而更好地提高成功率;其次,设计状态空间并定义离散化的动作和适当的奖励函数以引导无人机学习最优路径;最后在仿真环境中对DQN和Dueling DQN展开训练,结果表明:①Dueling DQN能规划出未知环境下从初始点到目标点的无碰撞路径,且能获得更高的奖励值;②经过50000次训练,Dueling DQN的成功率比DQN提高17.71%,碰撞率减少1.57%,超过最长步长率降低16.14%。

Associative Tasks Computing Offloading Scheme in Internet of Medical Things with Deep Reinforcement Learning

The Internet of Medical Things(Io MT) is regarded as a critical technology for intelligent healthcare in the foreseeable 6G era. Nevertheless, due to the limited computing power capability of edge devices and task-related coupling relationships, Io MT faces unprecedented challenges. Considering the associative connections among tasks, this paper proposes a computing offloading policy for multiple-user devices(UDs) considering device-to-device(D2D) communication and a multi-access edge computing(MEC)technique under the scenario of Io MT. Specifically,to minimize the total delay and energy consumption concerning the requirement of Io MT, we first analyze and model the detailed local execution, MEC execution, D2D execution, and associated tasks offloading exchange model. Consequently, the associated tasks’ offloading scheme of multi-UDs is formulated as a mixed-integer nonconvex optimization problem. Considering the advantages of deep reinforcement learning(DRL) in processing tasks related to coupling relationships, a Double DQN based associative tasks computing offloading(DDATO) algorithm is then proposed to obtain the optimal solution, which can make the best offloading decision under the condition that tasks of UDs are associative. Furthermore, to reduce the complexity of the DDATO algorithm, the cacheaided procedure is intentionally introduced before the data training process. This avoids redundant offloading and computing procedures concerning tasks that previously have already been cached by other UDs. In addition, we use a dynamic ε-greedy strategy in the action selection section of the algorithm, thus preventing the algorithm from falling into a locally optimal solution. Simulation results demonstrate that compared with other existing methods for associative task models concerning different structures in the Io MT network, the proposed algorithm can lower the total cost more effectively and efficiently while also providing a tradeoff between delay and energy consumption tolerance.

Convolutional Neural Network-Based Deep Q-Network (CNN-DQN) Resource Management in Cloud Radio Access Network

The recent surge of mobile subscribers and user data traffic has accelerated the telecommunication sector towards the adoption of the fifth-generation (5G) mobile networks. Cloud radio access network (CRAN) is a prominent framework in the 5G mobile network to meet the above requirements by deploying low-cost and intelligent multiple distributed antennas known as remote radio heads (RRHs). However, achieving the optimal resource allocation (RA) in CRAN using the traditional approach is still challenging due to the complex structure. In this paper, we introduce the convolutional neural network-based deep Q-network (CNN-DQN) to balance the energy consumption and guarantee the user quality of service (QoS) demand in downlink CRAN. We first formulate the Markov decision process (MDP) for energy efficiency (EE) and build up a 3-layer CNN to capture the environment feature as an input state space. We then use DQN to turn on/off the RRHs dynamically based on the user QoS demand and energy consumption in the CRAN. Finally, we solve the RA problem based on the user constraint and transmit power to guarantee the user QoS demand and maximize the EE with a minimum number of active RRHs. In the end, we conduct the simulation to compare our proposed scheme with nature DQN and the traditional approach.

基于DQN的多智能体深度强化学习运动规划方法

DQN方法作为经典的基于价值的深度强化学习方法,在多智能体运动规划等领域得到了广泛应用。然而,DQN方法面临一系列挑战,例如,DQN会过高估计Q值,计算Q值较为复杂,神经网络没有历史记忆能力,使用ε-greedy策略进行探索效率较低等。针对这些问题,提出了一种基于DQN的多智能体深度强化学习运动规划方法,该方法可以帮助智能体学习到高效稳定的运动规划策略,无碰撞地到达目标点。首先,在DQN方法的基础上,提出了基于Dueling的Q值计算优化机制,将Q值的计算方式改进为计算状态值和优势函数值,并根据当前正在更新的Q值网络的参数选择最优动作,使得Q值的计算更加简单准确;其次,提出了基于GRU的记忆机制,引入了GRU模块,使得网络可以捕捉时序信息,具有处理智能体历史信息的能力;最后,提出了基于噪声的有效探索机制,通过引入参数化的噪声,改变了DQN中的探索方式,提高了智能体的探索效率,使得多智能体系统达到探索-利用的平衡状态。在PyBullet仿真平台的6种不同的仿真场景中进行了测试,实验结果表明,所提方法可以使多智能体团队进行高效协作,无碰撞地到达各自目标点,且策略训练过程稳定。

基于改进DQN的移动机器人避障路径规划

针对一般强化学习方法下机器人在避障路径规划上学习时间长、探索能力差和奖励稀疏等问题,提出了一种基于改进深度Q网络(DQN)的移动机器人避障路径规划。首先在传统DQN算法基础上设计了障碍学习规则,避免对同一障碍重复学习,提升学习效率和成功率。其次提出奖励优化方法,利用状态间的访问次数差异给予奖励,平衡状态点的访问次数,避免过度访问;同时通过计算与目标点的欧氏距离,使其偏向于选择接近目标的路径,并取消远离目标惩罚,实现奖励机制的自适应优化。最后设计了动态探索因子函数,在后期训练中侧重利用强化学习策略选取动作和学习,提高算法性能和学习效率。实验仿真结果显示,与传统DQN算法相比,改进算法在训练时间上缩短了40.25%,避障成功率上提升了79.8%以及路径长度上缩短了2.25%,均体现了更好的性能。

面向无人艇的T-DQN智能避障算法研究

无人艇(Unmanned surface vehicle, USV)作为一种具有广泛应用前景的无人系统,其自主决策能力尤为关键.由于水面运动环境较为开阔,传统避障决策算法难以在量化规则下自主规划最优路线,而一般强化学习方法在大范围复杂环境下难以快速收敛.针对这些问题,提出一种基于阈值的深度Q网络避障算法(Threshold deep Q network, T-DQN),在深度Q网络(Deep Q network, DQN)基础上增加长短期记忆网络(Long short-term memory, LSTM)来保存训练信息,并设定经验回放池阈值加速算法的收敛.通过在不同尺度的栅格环境中进行实验仿真,实验结果表明, T-DQN算法能快速地收敛到最优路径,其整体收敛步数相比Q-learning算法和DQN算法,分别减少69.1%和24.8%,引入的阈值筛选机制使整体收敛步数降低41.1%.在Unity 3D强化学习仿真平台,验证了复杂地图场景下的避障任务完成情况,实验结果表明,该算法能实现无人艇的精细化避障和智能安全行驶.

超密集网络中基于改进DQN的接入选择算法

在超密集网络环境中,各个接入点密集部署在热点区域,构成了复杂的异构网络,用户需要选择接入合适的网络以获得最好的性能。如何为用户选择最优的网络,使用户自身或网络性能达到最佳,称为网络接入选择问题。为了解决超密集网络中用户的接入选择问题,综合考虑网络状态、用户偏好以及业务类型,结合负载均衡策略,提出了一种基于改进深度Q网络(deep Q network,DQN)的超密集网络接入选择算法。首先,通过分析网络属性和用户业务的偏好对网络选择的影响,选择合适的网络参数作为接入选择算法的参数;其次,将网络接入选择问题利用马尔可夫决策过程建模,分别对模型中的状态、动作和奖励函数进行设计;最后,利用DQN求解选网模型,得到最优选网策略。此外,为了避免DQN过高估计Q值,对传统DQN的目标函数进行优化,并且在训练神经网络时,引入了优先经验回放机制以提升学习效率。仿真结果表明,所提算法能够解决传统DQN的高估问题,加快神经网络的收敛,有效减少用户的阻塞,并改善网络的吞吐能力。

基于DQN算法的支线集装箱船航线规划与配载协同优化方法

针对支线集装箱船运输中喂给港数和靠泊条件不一,以及集装箱船队船型多样的特点,考虑航线规划与配载环节在实际运输过程中的紧密联系,研究支线集装箱船航线规划与配载协同优化方法。采用两阶段分层方法研究航线规划与集装箱配载问题,设置多个港口、不同船型及其贝位和堆栈组合、不同尺寸集装箱的集合,并确定其间基本关系,实现两阶段优化过程的完整性和连续性。第一阶段以航线总运营成本最小为目标建立船舶航线规划模型,第二阶段从主贝计划角度出发进行配载优化,确认集装箱与堆栈的对应关系,以船舶混装堆栈数最小为目标建立船舶配载模型,保证船舶稳性在航线任意时段均满足要求,并减少堆栈混装数量,提高到港作业效率。为实现模型高效求解,基于深度强化学习的Deep Q-learning Network(DQN)算法架构,设计了航线规划与配载决策对应的马尔可夫过程,结合问题自身特征分别完成强化学习智能体状态空间、动作空间以及奖励函数设计,构建了两阶段分层求解的DQN算法。实验结果表明:随着船舶数量和船舶装载率的增加,模型精确求解的时间大幅增加,部分算例无法在600 s内完成求解,而DQN算法可实现快速求解;与模型及粒子群优化(Particle Swarm Optimization,PSO)算法相比,DQN算法可高效求解不同规模下的算例,大规模算例求解最大耗时31.40 s,平均耗时30 s以内,求解效率较好;进一步计算表明,不同喂给港数量下PSO算法在求解时间上的平均标准差为11.20,而DQN算法平均标准差仅为1.74,鲁棒性更好。总体来看,DQN算法在求解时间上随问题规模变化而产生的波动较小,具有更加稳定的求解性能,可实现高效寻优。

基于Deep Q-Learning的抽取式摘要生成方法

为解决训练过程中需要句子级标签的问题,提出一种基于深度强化学习的无标签抽取式摘要生成方法,将文本摘要转化为Q-learning问题,并利用DQN(Deep Q-Network)学习Q函数。为有效表示文档,利用BERT(Bidirectional Encoder Representations from Transformers)作为句子编码器,Transformer作为文档编码器。解码器充分考虑了句子的信息富集度、显著性、位置重要性以及其与当前摘要之间的冗余程度等重要性等信息。该方法在抽取摘要时不需要句子级标签,可显著减少标注工作量。实验结果表明,该方法在CNN(Cable News Network)/DailyMail数据集上取得了最高的Rouge-L(38.35)以及可比较的Rouge-1(42.07)和Rouge-2(18.32)。

基于改进DQN算法的机器人路径规划

传统深度Q网络(DQN)算法通过融合深度神经网络和强化学习方法,解决了Q-learning算法在应对复杂环境时出现的维数灾难问题,被广泛应用于移动机器人的路径规划,但传统DQN算法的网络收敛速度较慢,路径规划效果较差,难以在较少的训练回合内获取最优路径。为了解决上述问题,提出一种改进的ERDQN算法。通过记录重复状态出现的频率,利用该频率重新计算Q值,使得在网络训练的过程中一种状态重复出现的次数越多,下一次出现该状态的概率越低,从而提高机器人对环境的探索能力,在一定程度上降低了网络收敛于局部最优的风险,减少了网络收敛的训练回合。根据机器人移动方向和机器人与目标点的距离,重新设计奖励函数。机器人在靠近目标点时能够获得正奖励,远离目标点时能够获得负奖励,并通过当前机器人的移动方向和机器人与目标点的距离调整奖励的绝对值,从而使机器人能够在避开障碍物的前提下规划出更优路径。实验结果表明,与DQN算法相比,ERDQN算法的平均得分提高了18.9%,规划出的路径长度和回合数减少了约20.1%和500。上述结果证明了ERDQN算法能够有效提高网络收敛速度及路径规划性能。

基于DQN的旋翼无人机着陆控制算法

针对无人机的着陆控制问题,研究了一种基于深度强化学习理论的旋翼无人机着陆控制算法。利用深度强化学习训练生成无人机智能体,根据观测结果给出动作指令,以实现自主着陆控制。首先,基于随机过程理论,将旋翼无人机的着陆控制问题转化为马尔可夫决策过程。其次,设计分别考虑无人机横向和纵向控制过程的奖励函数,将着陆控制问题转入强化学习框架。然后,采用深度Q网络(deep Q network,DQN)算法求解该强化学习问题,通过大量训练得到着陆控制智能体。最后,通过多种工况下的着陆平台进行大量的数值模拟和仿真分析,验证了算法的有效性。

基于输出层具有噪声的DQN的无人车路径规划

在DQN算法的框架下,研究了无人车路径规划问题.为提高探索效率,将处理连续状态的DQN算法加以变化地应用到离散状态,同时为平衡探索与利用,选择仅在DQN网络输出层添加噪声,并设计了渐进式奖励函数,最后在Gazebo仿真环境中进行实验.仿真结果表明:①该策略能快速规划出从初始点到目标点的无碰撞路线,与Q-learning算法、DQN算法和noisynet_DQN算法相比,该文提出的算法收敛速度更快;②该策略关于初始点、目标点、障碍物具有泛化能力,验证了其有效性与鲁棒性.

Walking Stability Control Method for Biped Robot on Uneven Ground Based on Deep Q-Network

A gait control method for a biped robot based on the deep Q-network (DQN) algorithm is proposed to enhance the stability of walking on uneven ground. This control strategy is an intelligent learning method of posture adjustment. A robot is taken as an agent and trained to walk steadily on an uneven surface with obstacles, using a simple reward function based on forward progress. The reward-punishment (RP) mechanism of the DQN algorithm is established after obtaining the offline gait which was generated in advance foot trajectory planning. Instead of implementing a complex dynamic model, the proposed method enables the biped robot to learn to adjust its posture on the uneven ground and ensures walking stability. The performance and effectiveness of the proposed algorithm was validated in the V-REP simulation environment. The results demonstrate that the biped robot's lateral tile angle is less than 3° after implementing the proposed method and the walking stability is obviously improved.

基于改进DQN燃气轮机转子故障诊断方法

燃气轮机转子系统作为燃气轮机关键部件,由于难以获取敏感故障特征导致故障诊断精度不高,影响设备的安全服役。针对以上问题,提出一种改进深度Q网络(DQN)深度强化学习燃气轮机转子系统故障诊断方法。首先,以采集的一维工况原始振动信号为输入,该DQN模型的环境状态采用故障样本集组成,转子故障类型为当前模型输入的动作集合;然后,DQN模型的智能体使用一维宽卷积神经网络(WDCNN)拟合得到Q网络,并使用ε-贪婪策略做出决策动作,反馈奖励和下一状态并存储到经验池内;智能体内采用时间差分误差(TD-error)优先经验回放,使得算法更加稳定和训练收敛;智能体与环境不断交互决策出最大奖励,输出最优策略故障诊断结果。将该模型应用于西储大学轴承数据集与燃气轮机试车台数据集中,分别达到99.2%与98.7%的准确率,可以用于快速有效地进行故障诊断。结果表明该改进DQN模型具有较高的故障诊断准确性与通用性。

Multi-Agent Deep Q-Networks for Efficient Edge Federated Learning Communications in Software-Defined IoT

Federated learning(FL)activates distributed on-device computation techniques to model a better algorithm performance with the interaction of local model updates and global model distributions in aggregation averaging processes.However,in large-scale heterogeneous Internet of Things(IoT)cellular networks,massive multi-dimensional model update iterations and resource-constrained computation are challenging aspects to be tackled significantly.This paper introduces the system model of converging softwaredefined networking(SDN)and network functions virtualization(NFV)to enable device/resource abstractions and provide NFV-enabled edge FL(eFL)aggregation servers for advancing automation and controllability.Multi-agent deep Q-networks(MADQNs)target to enforce a self-learning softwarization,optimize resource allocation policies,and advocate computation offloading decisions.With gathered network conditions and resource states,the proposed agent aims to explore various actions for estimating expected longterm rewards in a particular state observation.In exploration phase,optimal actions for joint resource allocation and offloading decisions in different possible states are obtained by maximum Q-value selections.Action-based virtual network functions(VNF)forwarding graph(VNFFG)is orchestrated to map VNFs towards eFL aggregation server with sufficient communication and computation resources in NFV infrastructure(NFVI).The proposed scheme indicates deficient allocation actions,modifies the VNF backup instances,and reallocates the virtual resource for exploitation phase.Deep neural network(DNN)is used as a value function approximator,and epsilongreedy algorithm balances exploration and exploitation.The scheme primarily considers the criticalities of FL model services and congestion states to optimize long-term policy.Simulation results presented the outperformance of the proposed scheme over reference schemes in terms of Quality of Service(QoS)performance metrics,including packet drop ratio,packet drop counts,packet delivery ratio,delay,and throughput.

基于深度Q神经网络(DQN)的空调冷却水系统无模型优化

在建筑空调水系统的优化控制领域,基于模型的控制方法得到了广泛的研究和验证。但基于模型的控制很大程度上依赖于精确的系统性能模型和足够的传感器,而这对于某些建筑来说是很难获得的。针对这一问题,本文提出了一种基于深度Q神经网络(DQN)的空调冷却水系统无模型优化方法,该方法以室外空气湿球温度、系统冷负荷及冷水机组开启状态为状态,以冷却塔风机和水泵的频率为动作,以系统性能系数(COP)为奖励。根据实际系统的实测数据进行建模,在模拟环境中使用基于粒子群优化算法的模型优化方法、基于Q值(Q learning)优化的强化学习方法和基于DQN的无模型优化方法进行实验,结果表明基于DQN的无模型优化方法的优化效果最好,有7.68%的平均COP提升与7.15%的节能率,在复杂系统下拥有较好的节能效果。