Cognitive interference decision method for air defense missile fuze based on reinforcement learning

To solve the problem of the low interference success rate of air defense missile radio fuzes due to the unified interference form of the traditional fuze interference system,an interference decision method based Q-learning algorithm is proposed.First,dividing the distance between the missile and the target into multiple states to increase the quantity of state spaces.Second,a multidimensional motion space is utilized,and the search range of which changes with the distance of the projectile,to select parameters and minimize the amount of ineffective interference parameters.The interference effect is determined by detecting whether the fuze signal disappears.Finally,a weighted reward function is used to determine the reward value based on the range state,output power,and parameter quantity information of the interference form.The effectiveness of the proposed method in selecting the range of motion space parameters and designing the discrimination degree of the reward function has been verified through offline experiments involving full-range missile rendezvous.The optimal interference form for each distance state has been obtained.Compared with the single-interference decision method,the proposed decision method can effectively improve the success rate of interference.

Resource Allocation for Cognitive Network Slicing in PD-SCMA System Based on Two-Way Deep Reinforcement Learning

In this paper,we propose the Two-way Deep Reinforcement Learning(DRL)-Based resource allocation algorithm,which solves the problem of resource allocation in the cognitive downlink network based on the underlay mode.Secondary users(SUs)in the cognitive network are multiplexed by a new Power Domain Sparse Code Multiple Access(PD-SCMA)scheme,and the physical resources of the cognitive base station are virtualized into two types of slices:enhanced mobile broadband(eMBB)slice and ultrareliable low latency communication(URLLC)slice.We design the Double Deep Q Network(DDQN)network output the optimal codebook assignment scheme and simultaneously use the Deep Deterministic Policy Gradient(DDPG)network output the optimal power allocation scheme.The objective is to jointly optimize the spectral efficiency of the system and the Quality of Service(QoS)of SUs.Simulation results show that the proposed algorithm outperforms the CNDDQN algorithm and modified JEERA algorithm in terms of spectral efficiency and QoS satisfaction.Additionally,compared with the Power Domain Non-orthogonal Multiple Access(PD-NOMA)slices and the Sparse Code Multiple Access(SCMA)slices,the PD-SCMA slices can dramatically enhance spectral efficiency and increase the number of accessible users.

Centralized Dynamic Spectrum Allocation in Cognitive Radio Networks Based on Fuzzy Logic and Q-Learning

A novel centralized approach for Dynamic Spectrum Allocation (DSA) in the Cognitive Radio (CR) network is presented in this paper. Instead of giving the solution in terms of formulas modeling network environment such as linear programming or convex optimization, the new approach obtains the capability of iteratively on-line learning environment performance by using Reinforcement Learning (RL) algorithm after observing the variability and uncertainty of the heterogeneous wireless networks. Appropriate decision-making access actions can then be obtained by employing Fuzzy Inference System (FIS) which ensures the strategy being able to explore the possible status and exploit the experiences sufficiently. The new approach considers multi-objective such as spectrum efficiency and fairness between CR Access Points (AP) effectively. By interacting with the environment and accumulating comprehensive advantages, it can achieve the largest long-term reward expected on the desired objectives and implement the best action. Moreover, the present algorithm is relatively simple and does not require complex calculations. Simulation results show that the proposed approach can get better performance with respect to fixed frequency planning scheme or general dynamic spectrum allocation policy.

Deep learning algorithm featuring continuous learning for modulation classifications in wireless networks

Although modulation classification based on deep neural network can achieve high Modulation Classification(MC)accuracies,catastrophic forgetting will occur when the neural network model continues to learn new tasks.In this paper,we simulate the dynamic wireless communication environment and focus on breaking the learning paradigm of isolated automatic MC.We innovate a research algorithm for continuous automatic MC.Firstly,a memory for storing representative old task modulation signals is built,which is employed to limit the gradient update direction of new tasks in the continuous learning stage to ensure that the loss of old tasks is also in a downward trend.Secondly,in order to better simulate the dynamic wireless communication environment,we employ the mini-batch gradient algorithm which is more suitable for continuous learning.Finally,the signal in the memory can be replayed to further strengthen the characteristics of the old task signal in the model.Simulation results verify the effectiveness of the method.

Multi-Objective Bacterial Foraging Optimization Algorithm Based on Effective Area in Cognitive Emergency Communication Networks

Cognitive emergency communication net-works can meet the requirements of large capac-ity,high density and low delay in emergency com-munications.This paper analyzes the properties of emergency users in cognitive emergency communi-cation networks,designs a multi-objective optimiza-tion and proposes a novel multi-objective bacterial foraging optimization algorithm based on effective area(MOBFO-EA)to maximize the transmission rate while maximizing the lifecycle of the network.In the algorithm,the effective area is proposed to prevent the algorithm from falling into a local optimum,and the diversity and uniformity of the Pareto-optimal solu-tions distributed in the effective area are used to eval-uate the optimization algorithm.Then,the dynamic preservation is used to enhance the competitiveness of excellent individuals and the uniformity and diversity of the Pareto-optimal solutions in the effective area.Finally,the adaptive step size,adaptive moving direc-tion and inertial weight are used to shorten the search time of bacteria and accelerate the optimization con-vergence.The simulation results show that the pro-posed MOBFO-EA algorithm improves the efficiency of the Pareto-optimal solutions by approximately 55%compared with the MOPSO algorithm and by approx-imately 60%compared with the MOBFO algorithm and has the fastest and smoothest convergence.

An Overview of Wireless Communication Technology Using Deep Learning

with the development of 5G,the future wireless communication network tends to be more and more intelligent.In the face of new service de-mands of communication in the future such as super-heterogeneous network,multiple communication sce-narios,large number of antenna elements and large bandwidth,new theories and technologies of intelli-gent communication have been widely studied,among which Deep Learning(DL)is a powerful technology in artificial intelligence(AI).It can be trained to con-tinuously learn to update the optimal parameters.This paper reviews the latest research progress of DL in in-telligent communication,and emphatically introduces five scenarios including Cognitive Radio(CR),Edge Computing(EC),Channel Measurement(CM),End to end Encoder/Decoder(EED)and Visible Light Com-munication(VLC).The prospect and challenges of further research and development in the future are also discussed.

Energy Efficient Transmission in Underlay CR-NOMA Networks Enabled by Reinforcement Learning

In order to improve the energy efficiency(EE)in the underlay cognitive radio(CR)networks,a power allocation strategy based on an actor-critic reinforcement learning is proposed,where a cluster of cognitive users(CUs)can simultaneously access to the same primary spectrum band under the interference constraints of the primary user(PU),by employing the non-orthogonal multiple access(NOMA)technique.In the proposed scheme,the optimization of the power allocation is formulated as a non-convex optimization problem.Additionally,the power allocation for different CUs is based on the actor-critic reinforcement learning model,in which the weighted data rate is set as the reward function,and the generated action strategy(i.e.the power allocation)is iteratively criticized and updated.Both the CU’s spectral efficiency and the PU’s interference constrains are considered in the training of the actor-critic reinforcement learning.Furthermore,the first order Taylor approximation as well as other manipulations are adopted to solve the power allocation optimization problem for the sake of considering the conventional channel conditions.According to the simulation results,we find that our scheme could achieve a higher spectral efficiency for the CUs compared to a benchmark scheme without learning process as well as the existing Q-learning based method,while the resultant interference affecting the PU transmission can be maintained at a given tolerated limit.

Predictive Spectrum Sensing Strategy Based on Reinforcement Learning

In this paper,we consider a cognitive radio(CR) system with a single secondary user(SU) and multiple licensed channels.The SU requests a fixed number of licensed channels and must sense the licensed channels one by one before transmission.By leveraging prediction based on correlation between the licensed channels,we propose a novel spectrum sensing strategy,to decide which channel is the best choice to sense in order to reduce the sensing time overhead and further improve the SU's achievable throughput.Since the correlation coefficients between the licensed channels cannot be exactly known in advance,the spectrum sensing strategy is designed based on the model-free reinforcement learning(RL).The experimental results show that the proposed spectrum sensing strategy based on reinforcement learning converges and outperforms random sensing strategy in terms of long-term statistics.

Towards a multi-agent reinforcement learning approach for joint sensing and sharing in cognitive radio networks

The adoption of the Fifth Generation(5G)and beyond 5G networks is driving the demand for learning approaches that enable users to co-exist harmoniously in a multi-user distributed environment.Although resource-constrained,the Cognitive Radio(CR)has been identified as a key enabler of distributed 5G and beyond networks due to its cognitive abilities and ability to access idle spectrum opportunistically.Reinforcement learning is well suited to meet the demand for learning in 5G and beyond 5G networks because it does not require the learning agent to have prior information about the environment in which it operates.Intuitively,CRs should be enabled to implement reinforcement learning to efficiently gain opportunistic access to spectrum and co-exist with each other.However,the application of reinforcement learning is straightforward in a single-agent environment and complex and resource intensive in a multi-agent and multi-objective learning environment.In this paper,(1)we present a brief history and overview of reinforcement learning and its limitations;(2)we provide a review of recent multi-agent learning methods proposed and multi-agent learning algorithms applied in Cognitive Radio(CR)networks;and(3)we further present a novel framework for multi-CR reinforcement learning and conclude with a synopsis of future research directions and recommendations.

DRL下UAV辅助认知无线电网络资源优化研究

认知无线电和能量收集技术为解决频谱利用率低和电池受限的问题提供了思路。为解决来自窃听者的安全威胁所导致的信息泄露问题,本文研究应用无人机协同干扰来增强多用户的物理层安全,以最大化安全速率。在底层模式下UAV辅助的能量采集认知无线电系统中,采用多智能体近端策略优化算法,结合长短期记忆网络增强序列样本数据的学习能力,提高算法的训练效率和有效性。仿真结果验证了所提方法的有效性和扩展性。

基于SumTree采样结合Double DQN的非合作式多用户动态功率控制方法

为了保证认知无线网络中次用户本身的通信服务质量,同时降低次用户因发射功率不合理而造成的功率损耗,提出了一种基于SumTree采样结合深度双Q网络(Double Deep Q Network,Double DQN)的非合作式多用户动态功率控制方法。通过这种方法,次用户可以不断与辅助基站进行交互,在动态变化的环境下经过不断的学习,选择以较低的发射功率完成功率控制任务。其次,该方法可以解耦目标Q值动作的选择和目标Q值的计算,能够有效减少过度估计和算法的损失。并且,在抽取经验样本时考虑到不同样本之间重要性的差异,采用了结合优先级和随机抽样的SumTree采样方法,既能保证优先级转移也能保证最低优先级的非零概率采样。仿真结果表明,该方法收敛后的算法平均损失值能稳定在0.04以内,算法的收敛速度也至少快了10个训练回合,还能提高次用户总的吞吐量上限和次用户功率控制的成功率,并且将次用户的平均功耗降低了0.5 mW以上。

面向天地融合网络的无线资源智能分配方法

为满足天地融合网络全时、全域通信需求,采用认知无线电技术可实现有限频谱资源的感知与高效利用,有效缓解同频干扰问题。文章提出了一种用于天地一体认知网络的信道选择和功率调整的无线资源智能分配方法,在保证主用户服务质量的前提下最大化系统数据速率。首先,将天地融合网络建模为异质图结构,通过用户距离估计信道状态信息,并且利用图卷积网络提取和分析关键环境特征。其次,采用深度强化学习探索底层拓扑环境信息,通过试错与奖惩机制不断优化资源分配策略。仿真结果验证了所提方法的收敛性,并且证明系统数据速率能够得到显著提升。

基于学习的能量采集认知M2M通信资源分配算法

本文针对能量采集认知机器到机器(Machine-to-Machine,M2M)通信的能量效率问题,在保证服务质量(Quality of Service,QoS)的条件下,提出了一种能效优化算法.以最大化网络中用户能效为目标,综合考虑传输功率控制、时隙分配、传输模式选择、中继选择以及每个设备的能量状态为约束,将优化问题建模为一个混合整数非线性规划问题.将该能效优化问题转化为离散时间有限状态马尔科夫决策过程(Discrete-time and Finite-state Markov Decision Process,DFMDP)进行求解.提出一种基于深度强化学习的算法寻找最优策略.仿真结果表明,所提算法在平均能效方面优于其他方案,且收敛速度在可接受范围内.

智能感知辅助的快速动态频谱抗干扰方法

机器学习赋能的动态频谱抗干扰方法能够通过频谱感知学习干扰规律,自主优化抗干扰策略,适应动态复杂的频谱环境。然而,现有大部分研究假设环境中仅有恶意干扰而不考虑其他通信系统的存在,且所提算法复杂度和算力要求较高。针对以上问题,考虑主用户和恶意干扰同时存在且用频规律未知的场景,设计了基于并行学习的智能感知算法和基于预先学习的动态频谱接入算法,无须通过随机探索频谱环境即可学习可用信道变化规律。仿真结果表明,所提算法在动态干扰攻击下能在线快速找到最优信道选择策略,在完美避开动态干扰的同时不对主用户造成无意干扰。

基于深度强化学习的认知物联网资源分配的策略研究

能量采集(Energy Harvesting,EH)和认知无线电(Cognitive Radio,CR)技术的组合可为物联网设备提供持续的能量,并有效地提高物联网系统的频谱效率。然而,在衬底模式下的认知物联网(Cognitive Radio IoT,CIoT)系统中,物联网设备之间的无线通信常常遭受窃听攻击。针对存在多窃听者条件下的CIoT系统无线通信场景,以保密速率作为系统保密性能指标。为解决所提的资源分配问题,将长短期记忆网络(Long-Term Memory Network,LSTM)、生成对抗网络(Generative Adversarial Networks,GAN)和深度强化学习(Deep Reinforcement Learning,DRL)算法相结合,设计一种联合能量采集时间和传输功率分配方案。数值仿真表明,与其他基准算法相比,所提方法能够有效地提高系统保密性能。

基于DRL的抗干扰电视频谱资源分配算法

将认知无线电与能量采集相结合,可以提高电视系统的频谱效率和能量效率。然而,由于无线信道的开放特性,频谱很容易受到恶意用户的干扰攻击,从而导致吞吐量下降。对此,将抗干扰频谱分配问题表述为没有任何先验知识的马尔可夫决策过程,然后提出一种基于深度确定性策略梯度(Deep Deterministic Policy Gradient,DDPG)的资源分配算法。在多种干扰环境下的仿真实验结果表明,该算法能够有效地减少恶意干扰带来的不利影响。

一种基于深度强化学习的抗干扰信道接入方法

认知无线电网络中,由于无线电环境的开放性和共享性,次级用户极易受到干扰,导致其吞吐量降低。而多个干扰机的出现会进一步急剧降低次级用户的吞吐量。针对这一问题,提出了一种在认知无线电网络中基于深度强化学习的抗干扰信道接入方法。首先,将多干扰环境下的信道接入建模为马尔科夫决策过程。然后,次级用户在没有先验信息条件下,以最大化吞吐量为目标,利用深度强化学习优化信道接入策略。所得到的信道接入策略使得次级用户在尽量不与主用户碰撞的前提下,自主选择一个干扰小的信道接入。仿真结果表明,与已有方法相比,所提方法将次级用户的吞吐量至少提升了19%。

认知无线网络中基于随机博弈框架的频率分配

为了解决认知无线网络中分布式的动态频率分配问题,采用随机博弈的框架,将认知链路建模成自私理性的智能体,并提出了一种以最大化平均Q函数为目标的多智能体学习算法—MAQ。通过MAQ学习,分布式的智能体可以实现间接的协商而不需要交互Q函数和回报值,因为智能体的决策过程需要考虑其他用户的决策。理论证明了MAQ学习算法的收敛性。仿真结果表明,MAQ算法的吞吐量性能接近中心式的学习算法,但是MAQ只需要较少的信息交互。

认知无线电网络中基于强化学习的智能信道选择算法

认知无线电系统不仅要具有自适应性,更应具备一定的智能性。该文将强化学习理论引入到认知无线电系统中,用于解决次用户在频谱感知过程中的信道选择问题,提出了一种基于强化学习的信道选择算法。该算法在未知主用户占用规律和动态特性的前提下,仅通过不断与环境进行交互学习,便能够引导次用户选择"较好"信道优先进行感知,使次用户吞吐量得到提高。仿真结果表明,相对于现有信道选择算法,所提算法可有效提高次用户的吞吐量,并且在主用户使用规律发生变化时,能够自动实现二次收敛,可作为认知无线电系统迈向智能化的一种尝试。

应急通信系统中基于认知无线电的动态频谱分配技术方案

本文讨论一种可实现的基于认知无线电的动态频谱分配技术的应急通信方案。在突发事故现场中,来自不同地方的救援力量拥有各自的集群网络。在此基础上,我们将每个集群网络视为认知无线电网中的一个认知节点,所有节点一起组成认知无线电网络。而认知无线电网络与强化学习系统有着密切的联系,故采用强化学习算法学习整个认知无线电网络,优化整个网络信道动态分配。