Transformer-Aided Deep Double Dueling Spatial-Temporal Q-Network for Spatial Crowdsourcing Analysis

With the rapid development ofmobile Internet,spatial crowdsourcing has becomemore andmore popular.Spatial crowdsourcing consists of many different types of applications,such as spatial crowd-sensing services.In terms of spatial crowd-sensing,it collects and analyzes traffic sensing data from clients like vehicles and traffic lights to construct intelligent traffic prediction models.Besides collecting sensing data,spatial crowdsourcing also includes spatial delivery services like DiDi and Uber.Appropriate task assignment and worker selection dominate the service quality for spatial crowdsourcing applications.Previous research conducted task assignments via traditional matching approaches or using simple network models.However,advanced mining methods are lacking to explore the relationship between workers,task publishers,and the spatio-temporal attributes in tasks.Therefore,in this paper,we propose a Deep Double Dueling Spatial-temporal Q Network(D3SQN)to adaptively learn the spatialtemporal relationship between task,task publishers,and workers in a dynamic environment to achieve optimal allocation.Specifically,D3SQNis revised through reinforcement learning by adding a spatial-temporal transformer that can estimate the expected state values and action advantages so as to improve the accuracy of task assignments.Extensive experiments are conducted over real data collected fromDiDi and ELM,and the simulation results verify the effectiveness of our proposed models.

Convolutional Neural Network-Based Deep Q-Network (CNN-DQN) Resource Management in Cloud Radio Access Network

The recent surge of mobile subscribers and user data traffic has accelerated the telecommunication sector towards the adoption of the fifth-generation (5G) mobile networks. Cloud radio access network (CRAN) is a prominent framework in the 5G mobile network to meet the above requirements by deploying low-cost and intelligent multiple distributed antennas known as remote radio heads (RRHs). However, achieving the optimal resource allocation (RA) in CRAN using the traditional approach is still challenging due to the complex structure. In this paper, we introduce the convolutional neural network-based deep Q-network (CNN-DQN) to balance the energy consumption and guarantee the user quality of service (QoS) demand in downlink CRAN. We first formulate the Markov decision process (MDP) for energy efficiency (EE) and build up a 3-layer CNN to capture the environment feature as an input state space. We then use DQN to turn on/off the RRHs dynamically based on the user QoS demand and energy consumption in the CRAN. Finally, we solve the RA problem based on the user constraint and transmit power to guarantee the user QoS demand and maximize the EE with a minimum number of active RRHs. In the end, we conduct the simulation to compare our proposed scheme with nature DQN and the traditional approach.

Walking Stability Control Method for Biped Robot on Uneven Ground Based on Deep Q-Network

A gait control method for a biped robot based on the deep Q-network (DQN) algorithm is proposed to enhance the stability of walking on uneven ground. This control strategy is an intelligent learning method of posture adjustment. A robot is taken as an agent and trained to walk steadily on an uneven surface with obstacles, using a simple reward function based on forward progress. The reward-punishment (RP) mechanism of the DQN algorithm is established after obtaining the offline gait which was generated in advance foot trajectory planning. Instead of implementing a complex dynamic model, the proposed method enables the biped robot to learn to adjust its posture on the uneven ground and ensures walking stability. The performance and effectiveness of the proposed algorithm was validated in the V-REP simulation environment. The results demonstrate that the biped robot's lateral tile angle is less than 3° after implementing the proposed method and the walking stability is obviously improved.

Multi-Agent Deep Q-Networks for Efficient Edge Federated Learning Communications in Software-Defined IoT

Federated learning(FL)activates distributed on-device computation techniques to model a better algorithm performance with the interaction of local model updates and global model distributions in aggregation averaging processes.However,in large-scale heterogeneous Internet of Things(IoT)cellular networks,massive multi-dimensional model update iterations and resource-constrained computation are challenging aspects to be tackled significantly.This paper introduces the system model of converging softwaredefined networking(SDN)and network functions virtualization(NFV)to enable device/resource abstractions and provide NFV-enabled edge FL(eFL)aggregation servers for advancing automation and controllability.Multi-agent deep Q-networks(MADQNs)target to enforce a self-learning softwarization,optimize resource allocation policies,and advocate computation offloading decisions.With gathered network conditions and resource states,the proposed agent aims to explore various actions for estimating expected longterm rewards in a particular state observation.In exploration phase,optimal actions for joint resource allocation and offloading decisions in different possible states are obtained by maximum Q-value selections.Action-based virtual network functions(VNF)forwarding graph(VNFFG)is orchestrated to map VNFs towards eFL aggregation server with sufficient communication and computation resources in NFV infrastructure(NFVI).The proposed scheme indicates deficient allocation actions,modifies the VNF backup instances,and reallocates the virtual resource for exploitation phase.Deep neural network(DNN)is used as a value function approximator,and epsilongreedy algorithm balances exploration and exploitation.The scheme primarily considers the criticalities of FL model services and congestion states to optimize long-term policy.Simulation results presented the outperformance of the proposed scheme over reference schemes in terms of Quality of Service(QoS)performance metrics,including packet drop ratio,packet drop counts,packet delivery ratio,delay,and throughput.

Reinforcement Learning with an Ensemble of Binary Action Deep Q-Networks

With the advent of Reinforcement Learning(RL)and its continuous progress,state-of-the-art RL systems have come up for many challenging and real-world tasks.Given the scope of this area,various techniques are found in the literature.One such notable technique,Multiple Deep Q-Network(DQN)based RL systems use multiple DQN-based-entities,which learn together and communicate with each other.The learning has to be distributed wisely among all entities in such a scheme and the inter-entity communication protocol has to be carefully designed.As more complex DQNs come to the fore,the overall complexity of these multi-entity systems has increased many folds leading to issues like difficulty in training,need for high resources,more training time,and difficulty in fine-tuning leading to performance issues.Taking a cue from the parallel processing found in the nature and its efficacy,we propose a lightweight ensemble based approach for solving the core RL tasks.It uses multiple binary action DQNs having shared state and reward.The benefits of the proposed approach are overall simplicity,faster convergence and better performance compared to conventional DQN based approaches.The approach can potentially be extended to any type of DQN by forming its ensemble.Conducting extensive experimentation,promising results are obtained using the proposed ensemble approach on OpenAI Gym tasks,and Atari 2600 games as compared to recent techniques.The proposed approach gives a stateof-the-art score of 500 on the Cartpole-v1 task,259.2 on the LunarLander-v2 task,and state-of-the-art results on four out of five Atari 2600 games.

UAV Autonomous Navigation for Wireless Powered Data Collection with Onboard Deep Q-Network

In a rechargeable wireless sensor network,utilizing the unmanned aerial vehicle(UAV)as a mobile base station(BS)to charge sensors and collect data effectively prolongs the network’s lifetime.In this paper,we jointly optimize the UAV’s flight trajectory and the sensor selection and operation modes to maximize the average data traffic of all sensors within a wireless sensor network(WSN)during finite UAV’s flight time,while ensuring the energy required for each sensor by wireless power transfer(WPT).We consider a practical scenario,where the UAV has no prior knowledge of sensor locations.The UAV performs autonomous navigation based on the status information obtained within the coverage area,which is modeled as a Markov decision process(MDP).The deep Q-network(DQN)is employed to execute the navigation based on the UAV position,the battery level state,channel conditions and current data traffic of sensors within the UAV’s coverage area.Our simulation results demonstrate that the DQN algorithm significantly improves the network performance in terms of the average data traffic and trajectory design.

Multi-Agent Path Planning Method Based on Improved Deep Q-Network in Dynamic Environments

The multi-agent path planning problem presents significant challenges in dynamic environments,primarily due to the ever-changing positions of obstacles and the complex interactions between agents’actions.These factors contribute to a tendency for the solution to converge slowly,and in some cases,diverge altogether.In addressing this issue,this paper introduces a novel approach utilizing a double dueling deep Q-network(D3QN),tailored for dynamic multi-agent environments.A novel reward function based on multi-agent positional constraints is designed,and a training strategy based on incremental learning is performed to achieve collaborative path planning of multiple agents.Moreover,the greedy and Boltzmann probability selection policy is introduced for action selection and avoiding convergence to local extremum.To match radar and image sensors,a convolutional neural network-long short-term memory(CNN-LSTM)architecture is constructed to extract the feature of multi-source measurement as the input of the D3QN.The algorithm’s efficacy and reliability are validated in a simulated environment,utilizing robot operating system and Gazebo.The simulation results show that the proposed algorithm provides a real-time solution for path planning tasks in dynamic scenarios.In terms of the average success rate and accuracy,the proposed method is superior to other deep learning algorithms,and the convergence speed is also improved.

End-to-End Autonomous Driving Through Dueling Double Deep Q-Network

Recent years have seen the rapid development of autonomous driving systems,which are typically designed in a hierarchical architecture or an end-to-end architecture.The hierarchical architecture is always complicated and hard to design,while the end-to-end architecture is more promising due to its simple structure.This paper puts forward an end-to-end autonomous driving method through a deep reinforcement learning algorithm Dueling Double Deep Q-Network,making it possible for the vehicle to learn end-to-end driving by itself.This paper firstly proposes an architecture for the end-to-end lane-keeping task.Unlike the traditional image-only state space,the presented state space is composed of both camera images and vehicle motion information.Then corresponding dueling neural network structure is introduced,which reduces the variance and improves sampling efficiency.Thirdly,the proposed method is applied to The Open Racing Car Simulator(TORCS)to demonstrate its great performance,where it surpasses human drivers.Finally,the saliency map of the neural network is visualized,which indicates the trained network drives by observing the lane lines.A video for the presented work is available online,https://youtu.be/76ciJ mIHMD8 or https://v.youku.com/v_show/id_XNDM4 ODc0M TM4NA==.html.

Deep Q-Network Based Dynamic Trajectory Design for UAV-Aided Emergency Communications

In this paper,an unmanned aerial vehicle(UAV)-aided wireless emergence communication system is studied,where a UAV is deployed to support ground user equipments(UEs)for emergence communications.We aim to maximize the number of the UEs served,the fairness,and the overall uplink data rate via optimizing the trajectory of UAV and the transmission power of UEs.We propose a deep Q-network(DQN)based algorithm,which involves the well-known deep neural network(DNN)and Q-learning,to solve the UAV trajectory prob-lem.Then,based on the optimized UAV trajectory,we further propose a successive convex approximation(SCA)based algorithm to tackle the power control problem for each UE.Numerical simulations demonstrate that the proposed DQN based algorithm can achieve considerable performance gain over the existing benchmark algorithms in terms of fairness,the number of UEs served and overall uplink data rate via optimizing UAV’s trajectory and power optimization.

A traffic-aware Q-network enhanced routing protocol based on GPSR for unmanned aerial vehicle ad-hoc networks

In dense traffic unmanned aerial vehicle(UAV)ad-hoc networks,traffic congestion can cause increased delay and packet loss,which limit the performance of the networks;therefore,a traffic balancing strategy is required to control the traffic.In this study,we propose TQNGPSR,a traffic-aware Q-network enhanced geographic routing protocol based on greedy perimeter stateless routing(GPSR),for UAV ad-hoc networks.The protocol enforces a traffic balancing strategy using the congestion information of neighbors,and evaluates the quality of a wireless link by the Q-network algorithm,which is a reinforcement learning algorithm.Based on the evaluation of each wireless link,the protocol makes routing decisions in multiple available choices to reduce delay and decrease packet loss.We simulate the performance of TQNGPSR and compare it with AODV,OLSR,GPSR,and QNGPSR.Simulation results show that TQNGPSR obtains higher packet delivery ratios and lower end-to-end delays than GPSR and QNGPSR.In high node density scenarios,it also outperforms AODV and OLSR in terms of the packet delivery ratio,end-to-end delay,and throughput.

Manufacturing Resource Scheduling Based on Deep Q-Network

To optimize machine allocation and task dispatching in smart manufacturing factories, this paper proposes a manufacturing resource scheduling framework based on reinforcement learning(RL). The framework formulates the entire scheduling process as a multi-stage sequential decision problem, and further obtains the scheduling order by the combination of deep convolutional neural network(CNN) and improved deep Q-network(DQN). Specifically, with respect to the representation of the Markov decision process(MDP), the feature matrix is considered as the state space and a set of heuristic dispatching rules are denoted as the action space. In addition, the deep CNN is employed to approximate the state-action values, and the double dueling deep Qnetwork with prioritized experience replay and noisy network(D3QPN2) is adopted to determine the appropriate action according to the current state. In the experiments, compared with the traditional heuristic method, the proposed method is able to learn high-quality scheduling policy and achieve shorter makespan on the standard public datasets.

Intelligent Voltage Control Method in Active Distribution Networks Based on Averaged Weighted Double Deep Q-network Algorithm

High penetration of distributed renewable energy sources and electric vehicles(EVs)makes future active distribution network(ADN)highly variable.These characteristics put great challenges to traditional voltage control methods.Voltage control based on the deep Q-network(DQN)algorithm offers a potential solution to this problem because it possesses humanlevel control performance.However,the traditional DQN methods may produce overestimation of action reward values,resulting in degradation of obtained solutions.In this paper,an intelligent voltage control method based on averaged weighted double deep Q-network(AWDDQN)algorithm is proposed to overcome the shortcomings of overestimation of action reward values in DQN algorithm and underestimation of action reward values in double deep Q-network(DDQN)algorithm.Using the proposed method,the voltage control objective is incorporated into the designed action reward values and normalized to form a Markov decision process(MDP)model which is solved by the AWDDQN algorithm.The designed AWDDQN-based intelligent voltage control agent is trained offline and used as online intelligent dynamic voltage regulator for the ADN.The proposed voltage control method is validated using the IEEE 33-bus and 123-bus systems containing renewable energy sources and EVs,and compared with the DQN and DDQN algorithms based methods,and traditional mixed-integer nonlinear program based methods.The simulation results show that the proposed method has better convergence and less voltage volatility than the other ones.

基于深度强化学习的空天地一体化网络资源分配算法

空天地一体化网络(SAGIN)通过提高地面网络的资源利用率可以有效满足多种业务类型的通信需求,然而忽略了系统的自适应能力和鲁棒性及不同用户的服务质量(QoS)。针对这一问题,该文提出在空天地一体化网络架构下,面向城区和郊区通信的深度强化学习(DRL)资源分配算法。基于第3代合作伙伴计划(3GPP)标准中定义的用户参考信号接收功率(RSRP),考虑地面同频干扰情况,以不同域中基站的时频资源作为约束条件,构建了最大化系统用户的下行吞吐量优化问题。利用深度Q网络(DQN)算法求解该优化问题时,定义了能够综合考虑用户服务质量需求、系统自适应能力及系统鲁棒性的奖励函数。仿真结果表明,综合考虑无人驾驶汽车,沉浸式服务及普通移动终端通信业务需求时,表征系统性能的奖励函数值在2 000次迭代下,相较于贪婪算法提升了39.1%;对于无人驾驶汽车业务,利用DQN算法进行资源分配后,相比于贪婪算法,丢包数平均下降38.07%,时延下降了6.05%。

考虑行为克隆的深度强化学习股票交易策略

为提高股票投资的收益并降低风险,将模仿学习中的行为克隆思想引入深度强化学习框架中设计股票交易策略。在策略设计过程中,将对决DQN深度强化学习算法和行为克隆进行结合,使智能体在自主探索的同时模仿事先构造的投资专家的决策。选择不同行业的股票进行数值实验,说明了所设计的交易策略在年化收益率、夏普比率和卡玛比率等收益与风险指标上优于对比策略。研究结果表明:将模仿学习与深度强化学习相结合可以使智能体同时具有探索和模仿能力,从而提高模型的泛化能力和策略的适用性。

基于深度强化学习的多自动导引车运动规划

为解决移动机器人仓储系统中的多自动导引车(AGV)无冲突运动规划问题,建立了Markov决策过程模型,提出一种新的基于深度Q网络(DQN)的求解方法。将AGV的位置作为输入信息,利用DQN估计该状态下采取每个动作所能获得的最大期望累计奖励,并采用经典的深度Q学习算法进行训练。算例计算结果表明,该方法可以有效克服AGV车队在运动中的碰撞问题,使AGV车队能够在无冲突的情况下完成货架搬运任务。与已有启发式算法相比,该方法求得的AGV运动规划方案所需要的平均最大完工时间更短。

基于数字孪生和深度强化学习的矿井超前液压支架自适应抗冲支护方法

受深部开采冲击地压等地质灾害扰动的影响,存在矿井超前支护系统自感知能力差、智能抗冲自适应能力弱、缺乏决策控制能力等问题。针对上述问题,提出了一种基于数字孪生和深度强化学习的矿井超前液压支架自适应抗冲支护方法。通过多源传感器感知巷道环境和超前液压支架支护状态,在虚拟世界中创建物理实体的数字孪生模型,其中物理模型精确展现超前液压支架的结构特征和细节,控制模型实现超前液压支架的自适应控制,机理模型实现对超前液压支架自适应支护的逻辑描述和机理解释,数据模型存储超前液压支架实体运行数据和孪生数据,仿真模型完成超前液压支架立柱仿真以实现超前液压支架与数字孪生模型虚实交互。根据基于深度Q网络(DQN)的超前液压支架自适应抗冲决策算法,对仿真环境中巷道抗冲支护进行智能决策,并依据决策结果对物理实体和数字孪生模型下达调控指令,实现超前液压支架智能控制。实验结果表明:立柱位移与压力变化一致,说明超前液压支架立柱仿真模型设计合理,从而验证了数字孪生模型的准确性;基于DQN的矿井超前液压支架自适应抗冲决策算法可通过调节液压支架控制器PID参数,自适应调控立柱压力,提升巷道安全等级,实现超前液压支架自适应抗冲支护。

一种分布式会议管理系统的设计与实现

基于现代会议管理的需求,设计和实现一种C/S和B/S混合部署的会议管理系统。包括会议管理服务中心和若干个会议现场,会议管理服务中心包括数据服务器、应用服务器、Web服务器、通信网关和出口路由器;会议现场包括若干个便携式电脑、RFID(Radio Frequency Identification)读卡器、二维码阅读器、信息显示发布设备、现场WLAN设备及用户终端。使用RIA(Rich Internet Application)技术优化了B/S界面,应用RFID对会议过程中的细节进行监控,借助SAAS(Software as a Service)模式实现会议管理按需配置和快速部署。该系统显著提高了会议管理效率。

一种基于DQN的去中心化优先级卸载策略

边缘计算(EC)可在网络边缘为用户提供低延迟、高响应的服务。因此,资源利用率高、时延低的任务卸载策略成为研究的热门方向。但大部分现有的任务卸载研究是基于中心化的架构,通过中心化设施制定卸载策略并进行资源调度,容易受到单点故障的影响,且会产生较多的能耗和较高的时延。针对以上问题,提出一种基于深度Q网络(DQN)的去中心化优先级(DP-DQN)卸载策略。首先,设置通信矩阵模拟现实中边缘服务器有限的通信状态;其次,通过对任务设定优先级,使任务可以在不同边缘服务器之间跳转,保证各边缘服务器均可以自主制定卸载策略,完成任务卸载的去中心化;最后,根据任务的跳转次数为任务分配更多的计算资源,提高资源利用效率和优化效果。为了验证所提策略的有效性,针对不同DQN下参数的收敛性能进行了研究对比,实验结果表明,在不同测试情景下,DP-DQN的性能均优于本地算法、完全贪婪算法和多目标任务卸载算法,性能可提升约11%~19%。

基于D3QN的火力方案优选方法

针对在多类弹药协同攻击地面工事类目标任务中火力方案优选效率低的问题,提出一种基于双层决斗DQN(dueling double deep Q network,D3QN)的火力方案优选方法。该方法将打击过程建模为马尔科夫决策过程(Markov decision processes,MDP),设计其状态空间和动作空间,设计综合奖励函数激励火力方案生成策略优化,使智能体通过强化学习框架对策略进行自主训练。仿真实验结果表明,该方法对地面工事类目标的火力方案进行决策,相较于传统启发式智能算法能够获得较优的火力方案,其计算效率和结果的稳定性相较于传统深度强化学习算法具有更明显的优势。

VEC中基于DRL的“端-多边-云”协作计算卸载算法

目的为了解决车载边缘计算中用户服务质量低以及边缘节点资源不足的问题,方法结合车载边缘计算和停车边缘计算技术,提出“端-多边-云”协作计算卸载模型,并设计基于DRL的协作计算卸载与资源分配算法(cooperative computation offloading and resource allocation algorithm based on DRL,DRL-CCORA)。首先,将路边停放车辆的算力构建成停车边缘服务器(parking edge server,PES),联合边缘节点为车辆任务提供计算服务,减轻边缘节点的负载;其次,将计算卸载与资源分配问题转化为马尔可夫决策过程模型,综合时延、能耗和服务质量构建奖励函数,并根据任务需要的计算资源、任务的最大容忍时延以及车辆到PES的距离对计算任务进行预分类处理,缩减问题的规模;最后,利用双深度Q网络(double deep q network,DDQN)算法获得计算卸载和资源分配的最优策略。结果结果表明,相较于对比算法,所提算法的用户总服务质量提高了6.25%,任务的完成率提高了10.26%,任务计算的时延和能耗分别降低了18.8%、5.26%。结论所提算法优化了边缘节点的负载,降低了任务完成的时延和能耗,提高了用户的服务质量。