Transformer-Aided Deep Double Dueling Spatial-Temporal Q-Network for Spatial Crowdsourcing Analysis

With the rapid development ofmobile Internet,spatial crowdsourcing has becomemore andmore popular.Spatial crowdsourcing consists of many different types of applications,such as spatial crowd-sensing services.In terms of spatial crowd-sensing,it collects and analyzes traffic sensing data from clients like vehicles and traffic lights to construct intelligent traffic prediction models.Besides collecting sensing data,spatial crowdsourcing also includes spatial delivery services like DiDi and Uber.Appropriate task assignment and worker selection dominate the service quality for spatial crowdsourcing applications.Previous research conducted task assignments via traditional matching approaches or using simple network models.However,advanced mining methods are lacking to explore the relationship between workers,task publishers,and the spatio-temporal attributes in tasks.Therefore,in this paper,we propose a Deep Double Dueling Spatial-temporal Q Network(D3SQN)to adaptively learn the spatialtemporal relationship between task,task publishers,and workers in a dynamic environment to achieve optimal allocation.Specifically,D3SQNis revised through reinforcement learning by adding a spatial-temporal transformer that can estimate the expected state values and action advantages so as to improve the accuracy of task assignments.Extensive experiments are conducted over real data collected fromDiDi and ELM,and the simulation results verify the effectiveness of our proposed models.

Reinforcement Learning with an Ensemble of Binary Action Deep Q-Networks

With the advent of Reinforcement Learning(RL)and its continuous progress,state-of-the-art RL systems have come up for many challenging and real-world tasks.Given the scope of this area,various techniques are found in the literature.One such notable technique,Multiple Deep Q-Network(DQN)based RL systems use multiple DQN-based-entities,which learn together and communicate with each other.The learning has to be distributed wisely among all entities in such a scheme and the inter-entity communication protocol has to be carefully designed.As more complex DQNs come to the fore,the overall complexity of these multi-entity systems has increased many folds leading to issues like difficulty in training,need for high resources,more training time,and difficulty in fine-tuning leading to performance issues.Taking a cue from the parallel processing found in the nature and its efficacy,we propose a lightweight ensemble based approach for solving the core RL tasks.It uses multiple binary action DQNs having shared state and reward.The benefits of the proposed approach are overall simplicity,faster convergence and better performance compared to conventional DQN based approaches.The approach can potentially be extended to any type of DQN by forming its ensemble.Conducting extensive experimentation,promising results are obtained using the proposed ensemble approach on OpenAI Gym tasks,and Atari 2600 games as compared to recent techniques.The proposed approach gives a stateof-the-art score of 500 on the Cartpole-v1 task,259.2 on the LunarLander-v2 task,and state-of-the-art results on four out of five Atari 2600 games.

UAV Autonomous Navigation for Wireless Powered Data Collection with Onboard Deep Q-Network

In a rechargeable wireless sensor network,utilizing the unmanned aerial vehicle(UAV)as a mobile base station(BS)to charge sensors and collect data effectively prolongs the network’s lifetime.In this paper,we jointly optimize the UAV’s flight trajectory and the sensor selection and operation modes to maximize the average data traffic of all sensors within a wireless sensor network(WSN)during finite UAV’s flight time,while ensuring the energy required for each sensor by wireless power transfer(WPT).We consider a practical scenario,where the UAV has no prior knowledge of sensor locations.The UAV performs autonomous navigation based on the status information obtained within the coverage area,which is modeled as a Markov decision process(MDP).The deep Q-network(DQN)is employed to execute the navigation based on the UAV position,the battery level state,channel conditions and current data traffic of sensors within the UAV’s coverage area.Our simulation results demonstrate that the DQN algorithm significantly improves the network performance in terms of the average data traffic and trajectory design.

Walking Stability Control Method for Biped Robot on Uneven Ground Based on Deep Q-Network

A gait control method for a biped robot based on the deep Q-network (DQN) algorithm is proposed to enhance the stability of walking on uneven ground. This control strategy is an intelligent learning method of posture adjustment. A robot is taken as an agent and trained to walk steadily on an uneven surface with obstacles, using a simple reward function based on forward progress. The reward-punishment (RP) mechanism of the DQN algorithm is established after obtaining the offline gait which was generated in advance foot trajectory planning. Instead of implementing a complex dynamic model, the proposed method enables the biped robot to learn to adjust its posture on the uneven ground and ensures walking stability. The performance and effectiveness of the proposed algorithm was validated in the V-REP simulation environment. The results demonstrate that the biped robot's lateral tile angle is less than 3° after implementing the proposed method and the walking stability is obviously improved.

Multi-Agent Deep Q-Networks for Efficient Edge Federated Learning Communications in Software-Defined IoT

Federated learning(FL)activates distributed on-device computation techniques to model a better algorithm performance with the interaction of local model updates and global model distributions in aggregation averaging processes.However,in large-scale heterogeneous Internet of Things(IoT)cellular networks,massive multi-dimensional model update iterations and resource-constrained computation are challenging aspects to be tackled significantly.This paper introduces the system model of converging softwaredefined networking(SDN)and network functions virtualization(NFV)to enable device/resource abstractions and provide NFV-enabled edge FL(eFL)aggregation servers for advancing automation and controllability.Multi-agent deep Q-networks(MADQNs)target to enforce a self-learning softwarization,optimize resource allocation policies,and advocate computation offloading decisions.With gathered network conditions and resource states,the proposed agent aims to explore various actions for estimating expected longterm rewards in a particular state observation.In exploration phase,optimal actions for joint resource allocation and offloading decisions in different possible states are obtained by maximum Q-value selections.Action-based virtual network functions(VNF)forwarding graph(VNFFG)is orchestrated to map VNFs towards eFL aggregation server with sufficient communication and computation resources in NFV infrastructure(NFVI).The proposed scheme indicates deficient allocation actions,modifies the VNF backup instances,and reallocates the virtual resource for exploitation phase.Deep neural network(DNN)is used as a value function approximator,and epsilongreedy algorithm balances exploration and exploitation.The scheme primarily considers the criticalities of FL model services and congestion states to optimize long-term policy.Simulation results presented the outperformance of the proposed scheme over reference schemes in terms of Quality of Service(QoS)performance metrics,including packet drop ratio,packet drop counts,packet delivery ratio,delay,and throughput.

Convolutional Neural Network-Based Deep Q-Network (CNN-DQN) Resource Management in Cloud Radio Access Network

The recent surge of mobile subscribers and user data traffic has accelerated the telecommunication sector towards the adoption of the fifth-generation (5G) mobile networks. Cloud radio access network (CRAN) is a prominent framework in the 5G mobile network to meet the above requirements by deploying low-cost and intelligent multiple distributed antennas known as remote radio heads (RRHs). However, achieving the optimal resource allocation (RA) in CRAN using the traditional approach is still challenging due to the complex structure. In this paper, we introduce the convolutional neural network-based deep Q-network (CNN-DQN) to balance the energy consumption and guarantee the user quality of service (QoS) demand in downlink CRAN. We first formulate the Markov decision process (MDP) for energy efficiency (EE) and build up a 3-layer CNN to capture the environment feature as an input state space. We then use DQN to turn on/off the RRHs dynamically based on the user QoS demand and energy consumption in the CRAN. Finally, we solve the RA problem based on the user constraint and transmit power to guarantee the user QoS demand and maximize the EE with a minimum number of active RRHs. In the end, we conduct the simulation to compare our proposed scheme with nature DQN and the traditional approach.

Intelligent Voltage Control Method in Active Distribution Networks Based on Averaged Weighted Double Deep Q-network Algorithm

High penetration of distributed renewable energy sources and electric vehicles(EVs)makes future active distribution network(ADN)highly variable.These characteristics put great challenges to traditional voltage control methods.Voltage control based on the deep Q-network(DQN)algorithm offers a potential solution to this problem because it possesses humanlevel control performance.However,the traditional DQN methods may produce overestimation of action reward values,resulting in degradation of obtained solutions.In this paper,an intelligent voltage control method based on averaged weighted double deep Q-network(AWDDQN)algorithm is proposed to overcome the shortcomings of overestimation of action reward values in DQN algorithm and underestimation of action reward values in double deep Q-network(DDQN)algorithm.Using the proposed method,the voltage control objective is incorporated into the designed action reward values and normalized to form a Markov decision process(MDP)model which is solved by the AWDDQN algorithm.The designed AWDDQN-based intelligent voltage control agent is trained offline and used as online intelligent dynamic voltage regulator for the ADN.The proposed voltage control method is validated using the IEEE 33-bus and 123-bus systems containing renewable energy sources and EVs,and compared with the DQN and DDQN algorithms based methods,and traditional mixed-integer nonlinear program based methods.The simulation results show that the proposed method has better convergence and less voltage volatility than the other ones.

基于深度强化学习的多自动导引车运动规划

为解决移动机器人仓储系统中的多自动导引车(AGV)无冲突运动规划问题,建立了Markov决策过程模型,提出一种新的基于深度Q网络(DQN)的求解方法。将AGV的位置作为输入信息,利用DQN估计该状态下采取每个动作所能获得的最大期望累计奖励,并采用经典的深度Q学习算法进行训练。算例计算结果表明,该方法可以有效克服AGV车队在运动中的碰撞问题,使AGV车队能够在无冲突的情况下完成货架搬运任务。与已有启发式算法相比,该方法求得的AGV运动规划方案所需要的平均最大完工时间更短。

基于深度强化学习和隐私保护的群智感知动态任务分配策略

在移动群智感知(Mobile Crowd Sensing,MCS)中,动态任务分配的结果对提高系统效率和确保数据质量至关重要。然而,现有的大部分研究在处理动态任务分配时,通常将其简化为二分匹配模型,该简化模型未充分考虑任务属性与工人属性对匹配结果的影响,同时忽视了工人位置隐私的保护问题。针对这些不足,文章提出一种基于深度强化学习和隐私保护的群智感知动态任务分配策略。该策略首先通过差分隐私技术为工人位置添加噪声,保护工人隐私;然后利用深度强化学习方法自适应地调整任务批量分配;最后使用基于工人任务执行能力阈值的贪婪算法计算最优策略下的平台总效用。在真实数据集上的实验结果表明,该策略在不同参数设置下均能保持优越的性能,同时有效地保护了工人的位置隐私。

通信受限条件下多无人机协同环境覆盖路径规划

多无人机协同覆盖旨在有效分配多个无人机任务,实现给定区域的快速、高效全覆盖。然而,在现实应用场景中常常因为无人机之间距离超出通信范围,信号传输受阻,导致无人机之间的协作和信息交互面临极大挑战。为此,提出一种基于Deep Q Networks(DQN)的多无人机路径规划方法。采用通信中断率和最大通信中断时间两个指标来评价路径质量,通过构建与指标相关的奖励函数,实现了无人机团队的自主路径决策。仿真实验表明,所提方法在最短路径上可以与传统优化算法效果保持一致,权衡路径下在增加20%路径长度的情况下可以降低80%通信中断率,在全通信路径下则可以实现100%的全过程连接通信,因此可以根据不同的通信环境生成高效覆盖所有环境节点的路径。

Associative Tasks Computing Offloading Scheme in Internet of Medical Things with Deep Reinforcement Learning

The Internet of Medical Things(Io MT) is regarded as a critical technology for intelligent healthcare in the foreseeable 6G era. Nevertheless, due to the limited computing power capability of edge devices and task-related coupling relationships, Io MT faces unprecedented challenges. Considering the associative connections among tasks, this paper proposes a computing offloading policy for multiple-user devices(UDs) considering device-to-device(D2D) communication and a multi-access edge computing(MEC)technique under the scenario of Io MT. Specifically,to minimize the total delay and energy consumption concerning the requirement of Io MT, we first analyze and model the detailed local execution, MEC execution, D2D execution, and associated tasks offloading exchange model. Consequently, the associated tasks’ offloading scheme of multi-UDs is formulated as a mixed-integer nonconvex optimization problem. Considering the advantages of deep reinforcement learning(DRL) in processing tasks related to coupling relationships, a Double DQN based associative tasks computing offloading(DDATO) algorithm is then proposed to obtain the optimal solution, which can make the best offloading decision under the condition that tasks of UDs are associative. Furthermore, to reduce the complexity of the DDATO algorithm, the cacheaided procedure is intentionally introduced before the data training process. This avoids redundant offloading and computing procedures concerning tasks that previously have already been cached by other UDs. In addition, we use a dynamic ε-greedy strategy in the action selection section of the algorithm, thus preventing the algorithm from falling into a locally optimal solution. Simulation results demonstrate that compared with other existing methods for associative task models concerning different structures in the Io MT network, the proposed algorithm can lower the total cost more effectively and efficiently while also providing a tradeoff between delay and energy consumption tolerance.

考虑行为克隆的深度强化学习股票交易策略

为提高股票投资的收益并降低风险,将模仿学习中的行为克隆思想引入深度强化学习框架中设计股票交易策略。在策略设计过程中,将对决DQN深度强化学习算法和行为克隆进行结合,使智能体在自主探索的同时模仿事先构造的投资专家的决策。选择不同行业的股票进行数值实验,说明了所设计的交易策略在年化收益率、夏普比率和卡玛比率等收益与风险指标上优于对比策略。研究结果表明:将模仿学习与深度强化学习相结合可以使智能体同时具有探索和模仿能力,从而提高模型的泛化能力和策略的适用性。

基于FL-MADQN算法的NR-V2X车载通信频谱资源分配

针对5G新空口-车联网(New Radio-Vehicle to Everything,NR-V2X)场景下车对基础设施(Vehicle to Infrastructure,V2I)和车对车(Vehicle to Vehicle,V2V)共享上行通信链路的频谱资源分配问题,提出了一种联邦-多智能体深度Q网络(Federated Learning-Multi-Agent Deep Q Network,FL-MADQN)算法.该分布式算法中,每个车辆用户作为一个智能体,根据获取的本地信道状态信息,以网络信道容量最佳为目标函数,采用DQN算法训练学习本地网络模型.采用联邦学习加快以及稳定各智能体网络模型训练的收敛速度,即将各智能体的本地模型上传至基站进行聚合形成全局模型,再将全局模型下发至各智能体更新本地模型.仿真结果表明:与传统分布式多智能体DQN算法相比,所提出的方案具有更快的模型收敛速度,并且当车辆用户数增大时仍然保证V2V链路的通信效率以及V2I链路的信道容量.

演化算法的DQN网络参数优化方法

为了解决DQN(Deep Q Network)在早期会出现盲目搜索、勘探利用不均并导致整个算法收敛过慢的问题,从探索前期有利于算法训练的有效信息获取与利用的角度出发,以差分演化(Differential Evolution)算法为例,提出了一种基于演化算法优化DQN网络参数以加快其收敛速度的方法(DE-DQN)。首先,将DQN的网络参数编码为演化个体;其次,分别采用“运行步长”和“平均回报”两种适应度函数评价方式;利用CartPole控制问题进行仿真对比,验证了两种评价方式的有效性。最后,实验结果表明,在智能体训练5 000代时所提出的改进算法,以“运行步长”为适应度函数时,在运行步长、平均回报和累计回报上分别提高了82.7%,18.1%和25.1%,并优于改进DQN算法;以“平均回报”为适应度函数时,在运行步长、平均回报和累计回报上分别提高了74.9%,18.5%和13.3%并优于改进DQN算法。这说明了DE-DQN算法相较于传统的DQN及其改进算法前期能获得更多有用信息,加快收敛速度。

一种分布式会议管理系统的设计与实现

基于现代会议管理的需求,设计和实现一种C/S和B/S混合部署的会议管理系统。包括会议管理服务中心和若干个会议现场,会议管理服务中心包括数据服务器、应用服务器、Web服务器、通信网关和出口路由器;会议现场包括若干个便携式电脑、RFID(Radio Frequency Identification)读卡器、二维码阅读器、信息显示发布设备、现场WLAN设备及用户终端。使用RIA(Rich Internet Application)技术优化了B/S界面,应用RFID对会议过程中的细节进行监控,借助SAAS(Software as a Service)模式实现会议管理按需配置和快速部署。该系统显著提高了会议管理效率。

基于CNN-LSTM的MIMO-OFDM信号盲调制识别算法

无线通信信号的盲调制识别技术作为非协作通信的核心技术之一,在提高频谱利用效率以及未知信号解调中起着至关重要的作用。另外,非协作通信中存在着电磁环境未知,噪声干扰严重,信噪比低等问题,因此在非协作通信下进行未知信号的盲调制识别较为困难。为解决非协作通信中多输入多输出正交频分复用(MultipleInput Multiple-Output Orthogonal Frequency Division Multiplexing, MIMO-OFDM)信号在低信噪比下子载波盲调制识别的问题,本文使用CNN(Convolutional Neural Network,CNN)网络与LSTM(Long Short-Term Memory,LSTM)网络构建一维CNN-LSTM网络进行盲调制识别。鉴于I/Q数据具有很强特征表达能力,该算法选取I/Q数据作为第一输入特征直接输入网络。同时为了弥补噪声对I/Q数据的干扰,本文还选用抗噪声能力强的循环谱作为另一输入特征,为进一步提升循环谱的抗噪声能力,本文对循环谱进行切片累加操作得到抗噪声性能更好的循环谱切片累加序列作为第二输入特征。仿真结果表明,本文所提方法可以在SNR=2 dB条件下实现对{BPSK,QPSK,8PSK,16QAM,32QAM,128QAM}调制方式的识别,并且识别精度达到98%。

Q学习博弈论的WSNs混合覆盖漏洞恢复

针对恶劣环境下分布式无线传感器网络,为了降低成本与恢复能力,提出了一种Q学习博弈论的无线传感器网络混合覆盖漏洞恢复方法。首先设计了一种能够以分散、动态和自治的方式缩小覆盖差距的混合算法,该方法利用基于Q学习算法的博弈论概念,融合了节点重新定位和功率传输调整两种覆盖控制方案。对于所制定的潜在博弈论,传感器节点可以仅使用局部熟悉来恢复覆盖漏洞,从而减小覆盖间隙,每个传感器节点选择节点重新定位和调整感知范围。最后仿真结果表明,这里的提出的方法能够在存在连续随机覆盖漏洞条件下保持网络的整体覆盖。

基于UPLC-Q-TOF-MS/MS联合网络药理学及分子对接研究五味清浊颗粒治疗腹泻药效物质基础和作用机制

目的基于UPLC-Q-TOF-MS/MS技术鉴定五味清浊颗粒中的化学成分,联合网络药理学、分子对接技术探讨其治疗腹泻药效物质基础和作用机制。方法采用Eclipsepius C18色谱柱(50 mm×2.1 mm,1.8μm),以0.1%甲酸水溶液(A)-乙腈(B)为流动相进行梯度洗脱,柱温为30℃,流速为0.4 mL·min^(-1)。质谱采用电喷雾离子源(ESI)正、负离子模式,扫描范围m/z 50~2000条件下采集多级质谱碎片信息。应用网络药理学构建“核心成分-作用靶点-通路”的网络,对其潜在药效物质基础进行预测。利用AutoDock Vina进行分子对接验证。结果共鉴定出86个主要化学成分,包括生物碱类25个、黄酮类23个、有机酸类12个、鞣质类16个、苯丙素类2个、其他类化合物8个。网络药理学分析结果显示,槲皮素、木犀草素、鞣花酸、胡椒碱、山柰酚、荜茇宁主要作用于IL-6、TNF、EGFR、IFNG、IL-10、IL-8等核心靶点,调节PI3K-Akt、HIF-1、JAK-STAT等关键信号通路来发挥治疗腹泻作用。分子对接结果显示核心成分与核心靶点间具有良好的结合性能。结论该研究成功采用UPLC-Q-TOF-MS/MS技术对五味清浊颗粒化学成分进行全面分析鉴定,初步阐明其治疗腹泻的作用机制,为其药效物质基础和质量控制奠定基础。

基于强化学习的多对多拦截目标分配方法

针对空中对抗环境中多对多拦截的武器目标分配问题,提出了一种基于强化学习的多目标智能分配方法。在多对多拦截交战场景下,基于交战态势评估构建了目标分配的数学模型。通过引入目标威胁程度和拦截有效程度的概念,充分反映了各目标的拦截紧迫性和各拦截器的拦截能力表征,从而全面评估了攻防双方的交战态势。在目标分配模型的基础上,将目标分配问题构建为马尔可夫决策过程,并采用基于深度Q网络的强化学习算法训练求解。依靠环境交互下的自学习和奖励机制,有效实现了最优分配方案的动态生成。通过数学仿真构建多对多拦截场景,并验证了该方法的有效性,经训练后的目标分配方法能够满足多对多拦截中连续动态的任务分配要求。

面向城市道路的智能网联汽车多车道轨迹优化方法

为提高城市路网下智能网联汽车的通行效率以及燃油效率,提出面向城市道路的多车道时空轨迹优化方法。首先,结合多车道时空位置关系定义智能网联汽车状态与约束,综合考虑通行效率与燃油经济性构建时空轨迹复合优化模型,并采用庞特里亚金极大值算法进行求解。然后,本文设定协同换道的规则,并通过Q-learning算法获取最优的换道策略。最后,通过SUMO/Python联合仿真验证了该方法可以在不同车辆饱和程度、绿信比状态及最低通行速度条件下有效提高通行效率,且燃油效率得到明显改善。