Cooperative Channel Assignment for VANETs Based on Dual Reinforcement Learning

Dynamic channel assignment(DCA)is significant for extending vehicular ad hoc network(VANET)capacity and mitigating congestion.However,the un-known global state information and the lack of centralized control make channel assignment performances a challenging task in a distributed vehicular direct communication scenario.In our preliminary field test for communication under V2X scenario,we find that the existing DCA technology cannot fully meet the communication performance requirements of VANET.In order to improve the communication performance,we firstly demonstrate the feasibility and potential of reinforcement learning(RL)method in joint channel selection decision and access fallback adaptation design in this paper.Besides,a dual reinforcement learning(DRL)-based cooperative DCA(DRL-CDCA)mechanism is proposed.Specifically,DRL-CDCA jointly optimizes the decision-making behaviors of both the channel selection and back-off adaptation based on a multi-agent dual reinforcement learning framework.Besides,nodes locally share and incorporate their individual rewards after each communication to achieve regional consistency optimization.Simulation results show that the proposed DRL-CDCA can better reduce the one-hop packet delay,improve the packet delivery ratio on average when compared with two other existing mechanisms.

Multi-Agent Few-Shot Meta Reinforcement Learning for Trajectory Design and Channel Selection in UAV-Assisted Networks

Unmanned aerial vehicle(UAV)-assisted communications have been considered as a solution of aerial networking in future wireless networks due to its low-cost, high-mobility, and swift features. This paper considers a UAV-assisted downlink transmission,where UAVs are deployed as aerial base stations to serve ground users. To maximize the average transmission rate among the ground users, this paper formulates a joint optimization problem of UAV trajectory design and channel selection, which is NP-hard and non-convex. To solve the problem, we propose a multi-agent deep Q-network(MADQN) scheme.Specifically, the agents that the UAVs act as perform actions from their observations distributively and share the same reward. To tackle the tasks where the experience is insufficient, we propose a multi-agent meta reinforcement learning algorithm to fast adapt to the new tasks. By pretraining the tasks with similar distribution, the learning model can acquire general knowledge. Simulation results have indicated the MADQN scheme can achieve higher throughput than fixed allocation. Furthermore, our proposed multiagent meta reinforcement learning algorithm learns the new tasks much faster compared with the MADQN scheme.

A Deep Reinforcement Learning-Based Technique for Optimal Power Allocation in Multiple Access Communications

Formany years,researchers have explored power allocation(PA)algorithms driven bymodels in wireless networks where multiple-user communications with interference are present.Nowadays,data-driven machine learning methods have become quite popular in analyzing wireless communication systems,which among them deep reinforcement learning(DRL)has a significant role in solving optimization issues under certain constraints.To this purpose,in this paper,we investigate the PA problem in a k-user multiple access channels(MAC),where k transmitters(e.g.,mobile users)aim to send an independent message to a common receiver(e.g.,base station)through wireless channels.To this end,we first train the deep Q network(DQN)with a deep Q learning(DQL)algorithm over the simulation environment,utilizing offline learning.Then,the DQN will be used with the real data in the online training method for the PA issue by maximizing the sumrate subjected to the source power.Finally,the simulation results indicate that our proposedDQNmethod provides better performance in terms of the sumrate compared with the available DQL training approaches such as fractional programming(FP)and weighted minimum mean squared error(WMMSE).Additionally,by considering different user densities,we show that our proposed DQN outperforms benchmark algorithms,thereby,a good generalization ability is verified over wireless multi-user communication systems.

Reinforcement Learning-Based Handover Scheme with Neighbor Beacon Frame Transmission

Mobility support to change the connection from one access point(AP)to the next(i.e.,handover)becomes one of the important issues in IEEE 802.11 wireless local area networks(WLANs).During handover,the channel scanning procedure,which aims to collect neighbor AP(NAP)information on all available channels,accounts for most of the delay time.To reduce the channel scanning procedure,a neighbor beacon frame transmission scheme(N-BTS)was proposed for a seamless handover.N-BTS can provide a seamless handover by removing the channel scanning procedure.However,N-BTS always requires operating overhead even if there are few mobile stations(MSs)for the handover.Therefore,this paper proposes a reinforcement learning-based handover scheme with neighbor beacon frame transmission(MAN-BTS)to properly consider the use of N-BTS.The optimization equation is defined to maximize the expected reward tofind the optimal policy and is solved using Q-learning.Simulation results show that the proposed scheme outperforms the comparison schemes in terms of the expected reward.

Distributed Multi-Cell Multi-User MISO Downlink Beamforming via Deep Reinforcement Learning

The sum rate maximization beamforming problem for a multi-cell multi-user multiple-input single-output interference channel(MISO-IC)system is considered.Conventionally,the centralized and distributed beamforming solutions to the MISO-IC system have high computational complexity and bear a heavy burden of channel state information exchange between base stations(BSs),which becomes even much worse in a large-scale antenna system.To address this,we propose a distributed deep reinforcement learning(DRL)based approach with lim⁃ited information exchange.Specifically,the original beamforming problem is decomposed of the problems of beam direction design and power allocation and the costs of information exchange between BSs are significantly reduced.In particular,each BS is provided with an inde⁃pendent deep deterministic policy gradient network that can learn to choose the beam direction scheme and simultaneously allocate power to users.Simulation results illustrate that the proposed DRL-based approach has comparable sum rate performance with much less information exchange over the conventional distributed beamforming solutions.

注浆加固砖砌体的受压性能试验研究

我国许多20世纪所修建的砖石砌体结构排水管渠面临年久失修、功能丧失等问题,影响城市的正常运作,亟需对其进行加固修缮。针对砌体结构排水管渠长期遭地下水冲刷、灰缝砂浆不饱满、性能劣化的特点,可采用注浆法填充原结构灰缝,在充分修复原结构的基础上再进行整体结构加固修复。为了验证注浆加固受损砌体的可行性,通过砌筑灰缝饱满度为50%~60%的砌体试件来模拟砌体灰缝砂浆长期受水流冲刷的状态,并完成6个注浆砖砌体、3个未注浆砖砌体的受压试验来对比分析注浆前后砖砌体的受力性能,根据试验结果,拟合砖砌体应力-应变曲线。结果表明,注浆后砖砌体的抗压强度为未注浆砌体的280%,注浆加固砌体的弹性模量和泊松比为未注浆砌体的281%、60%。证明注浆法能够改善受损砌体结构的力学性能,在砖石砌体结构排水管渠的加固工程中可起到修复原砌体结构的作用。

槽型套管加强方钢管X形节点平面内受弯性能试验

槽型套管适用于在役钢管结构的节点局部加强,可有效提高节点承载力和刚度。为研究槽型套管对方钢管节点平面内受弯性能的加强效果,对5组支主管宽度比β为0.58、0.7和1.0的试件进行平面内受弯试验。试验表明槽型套管加强可以有效提升节点受弯性能,β=0.58和β=0.7时槽型套管加强节点的受弯承载力分别提高37%和163%;加强节点的刚度略低于等宽节点,但其延性更优;支主管宽度比和槽型套管厚度是影响加强节点平面内受弯性能的主要参数。槽型套管加强可有效保护节点主管,使试件失效模式由主管或焊缝破坏转移为支管破坏。进一步机理分析表明,加强套管可与主管管壁形成共同承载体,通过局部增厚作用提高节点承载力和刚度。据此对比分析了该类加强节点受弯承载力计算公式的适用性,可供设计参考。

补肾健脾法对OP大鼠肌骨中ULK1/FUNDC1介导的线粒体自噬的影响

目的 探究补肾健脾法对骨质疏松(osteoporosis, OP)大鼠肌骨中ULK1/FUNDC1介导的线粒体自噬的影响。方法 72只2月龄雄性SD大鼠随机分为正常组、模型组、补肾组、健脾组、补肾健脾组、西药组(福善美),每组12只,模型组及各个用药组采用尾部悬吊法进行造模,造模时间为5周。正常组与模型组予以超纯水灌胃,其余各组予以相应药物进行灌胃。造模结束后行腹主动脉采血,酶联免疫吸附法(ELISA)测定血清碱性磷酸酶(ALP)、抗酒石酸酸性磷酸酶(TRACP)含量;计算机微断层扫描(Micro-CT)检测大鼠左侧股骨远端形态学变化;双能X线行股骨骨密度(bone mineral density, BMD)检测;马松(Masson)染色观察腓肠肌病理变化;蛋白免疫印迹法(Western blot)检测股骨、腓肠肌中线粒体蛋白ULK1、FUNDC1、LC3-Ⅱ/LC3-Ⅰ蛋白表达。结果 与正常组相较,模型组ALP(P<0.01)降低、TRACP(P<0.01)升高,骨小梁面积、BMD(P<0.01)均下降,提示造模成功。Masson染色提示在中医不同治法干预下腓肠肌组织纤维化程度得到缓解。Western blot结果显示,在中医不同治法干预下与模型组股骨相比ULK1、LC3-Ⅱ/LC3-Ⅰ、FUNDC1表达降低(P<0.05),线粒体自噬水平降低;与模型组腓肠肌相比ULK1、FUNDC1、LC3-Ⅱ/LC3-Ⅰ蛋白表达升高(P<0.05),线粒体自噬水平升高。结论 补肾健脾中医治法对于OP模型大鼠的治疗效果可能是通过ULK1/FUNDC1通路调控线粒体自噬发挥作用,综合来看补肾健脾法效果较优。

骨水泥钉道强化椎体对退行性腰椎侧弯椎间融合器沉降及融合的影响

目的探讨骨水泥钉道强化椎体对合并骨质疏松的退行性腰椎侧弯患者术后椎间融合器沉降及融合的影响。方法回顾性分析2020年1月至2023年6月收治的退行性腰椎侧弯患者24例,按手术方式分为A组(前外侧入路腰椎融合术联合后路经皮椎弓根螺钉内固定,16例)和B组(前外侧入路腰椎融合术联合后路经皮椎弓根螺钉内固定并加行骨水泥强化,8例),观察两组患者术中终板损伤、术后融合器沉降及融合、螺钉松动等情况,评估手术前后疼痛视觉模拟评分(VAS)、Oswestry功能障碍指数(ODI)。结果两组患者术后VAS评分、ODI指数明显下降,且较术前差异具有统计学意义(P<0.05)。B组术中终板损伤发生率、融合器融合率及沉降率、螺钉松动发生率较A组下降,但差异不具有统计学意义(P>0.05)。结论骨水泥钉道强化不仅可以增强椎弓根螺钉的把持力及抗拔出能力,同时可以增强椎体强度,可能对降低融合器沉降及融合有积极作用。

基于自适应p持续的移动自组网信道接入和资源分配算法

针对基于p持续的移动自组网(MANET)信道接入和资源分配问题,提出一种具有低复杂度的自适应信道接入和资源分配算法。首先,考虑到自组网无中心分布式组网特点,以每个节点的信道利用率最大化为目标建立优化问题;其次将该问题建模为马尔可夫决策过程并定义状态、动作和奖励函数;最后基于策略梯度训练网络参数,联合优化竞争概率、优先级增长因子以及通信节点数量。仿真实验结果表明,所提算法可以显著提高p-持续载波侦听多址接入(CSMA)协议的性能,与固定竞争概率和p值预定义的方案相比,所提算法的信道利用率提高了45%和17%;此外,当节点数量小于35时,所提算法优于固定接入节点数量的方案。同时,在节点数据包到达率较高时,所提算法可以充分利用信道,减少时隙资源浪费。

低时延网络多路通信数据无冲突调度仿真

随着信息传播数据量的飞速增长,以及低时延网络的广泛应用,低时延网络无冲突调度的需求也进一步增加。为解决当前通信调度算法时效性差、调度效果不佳等问题,提出了基于强化学习演员-评论家算法的无冲突调度模型。模型首先对低时延网络架构进行分析,并对状态空间中的多路通信网络进行数据监测;然后基于受限制的马尔科夫决策对其进行无冲突调度,并基于贝尔曼方程计算结果进行策略调度;最后使用强化学习中的Actor-Critic算法对调度算法进行优化,通过Actor提出调度方案,通过Critic评价调度情况,进而达到无冲突调度的效果。实验结果表明,所提算法将冲突率降低了8.72%,将吞吐率提升了7.58%,降低了数据在传输过程中丢失和乱码的概率,提高了通信数据的传输效率。

新建航道基坑下穿高铁桥梁对邻近桥墩的影响分析

[目的]新建航道基坑下穿高铁桥梁时,基坑开挖卸载会引起桥梁周边土体产生位移,进而影响高铁桥梁。为确保高铁线路的运营安全,需对新建航道基坑下穿高铁桥梁时对邻近桥墩的影响进行分析。[方法]以京杭大运河新建航道基坑下穿沪昆高铁工程为背景,基于有限元模拟和现场实测,分析了航道基坑施工对既有高铁桥墩的影响,评估了航道基坑工程采用多排钻孔桩+斜撑加固方案的可行性和效果。[结果及结论]航道基坑施工主要会引起高铁桥墩产生顺桥向的水平变形,建议采取措施适当增大顺桥向刚度;钻孔桩越靠近高铁桥墩,其水平位移越大,随着航道基坑的施工,钻孔桩水平位移呈现先增大后减小的趋势;采取多排钻孔桩+斜撑的加固方案后,桥墩变形小于相应规范规定的2 mm限值要求,满足列车安全运营的要求。

Cooperative channel assignment for VANETs based on multiagent reinforcement learning

Dynamic channel assignment(DCA)plays a key role in extending vehicular ad-hoc network capacity and mitigating congestion.However,channel assignment under vehicular direct communication scenarios faces mutual influence of large-scale nodes,the lack of centralized coordination,unknown global state information,and other challenges.To solve this problem,a multiagent reinforcement learning(RL)based cooperative DCA(RLCDCA)mechanism is proposed.Specifically,each vehicular node can successfully learn the proper strategies of channel selection and backoff adaptation from the real-time channel state information(CSI)using two cooperative RL models.In addition,neural networks are constructed as nonlinear Q-function approximators,which facilitates the mapping of the continuously sensed input to the mixed policy output.Nodes are driven to locally share and incorporate their individual rewards such that they can optimize their policies in a distributed collaborative manner.Simulation results show that the proposed multiagent RL-CDCA can better reduce the one-hop packet delay by no less than 73.73%,improve the packet delivery ratio by no less than 12.66%on average in a highly dense situation,and improve the fairness of the global network resource allocation.

基于多智能体近端策略优化的多信道动态频谱接入

为了在多用户多信道通信场景中应用动态频谱接入(Dynamic Spectrum Access,DSA)技术提高通信效率,保证用户公平,本文基于多智能体近端策略优化(Multi-Agent Proximal Policy Optimization,MAPPO)提出了MAPPO-DSA算法.该算法首先针对单信道接入在多个信道同时空闲时存在的频谱浪费问题,使用多信道接入作为解决方案.同时,多信道接入导致状态空间与动作空间指数增长,计算成本高,学习难度大.为此本文引入MAPPO深度强化学习(Deep Reinforcement Learning,DRL)算法,在复杂环境中高效学习和优化接入策略.通过设计优化MAPPO中观测及奖励等强化学习要素和共享网络参数来保证用户公平.最后,在不同场景下的实验结果表明,所提出的MAPPO-DSA能够学习到近似最优的接入策略,部分场景中的网络吞吐量逼近理论上限,显著优于现有算法,且有效保证用户公平.

基于深度强化学习的通信抗干扰系统

由于电磁信道的开放特征,存在恶意节点对正常数据流实施干扰,阻止合法接收者获得信息,进而截获并篡改数据,因此针对通信抗干扰的研究非常重要。传统的抗干扰方法采用单一抗干扰方式,并不能根据环境自适应改变抗干扰策略,因此难以在复杂通信环境中达到较好的抗干扰效果。为应对这些挑战,研究了基于深度强化学习(Deep Reinforcement Learning,DRL)的抗干扰通信模型,并搭建仿真系统,利用DRL算法实现智能抗干扰决策。仿真结果表明,提出的智能抗干扰决策能够在复杂环境下根据环境选择最优抗干扰方案,有效提升通信质量。

槽钢加固上海市南市区图书馆旧址整体移位数值分析及其关键部位变形监测

石库门里弄住宅是近现代重要史迹及沪上代表性建筑,展示着特定时期城市文化的沉淀。以上海市南市区图书馆旧址工程为例,采用槽钢内撑外拉的方式对整体结构进行了加固,利用既有建筑整体移位技术对该砖-木混合结构进行了顶升和平移,采用位移传感器监测了结构在平移过程中关键部位的位移响应;采用ABAQUS有限元软件建立了上海市南市区图书馆旧址工程的整体有限元模型,得到了结构在顶升、平移过程中关键部位的位移响应,并将模拟结果与实际位移监测结果进行了对比;基于验证的有限元模型,开展了参数分析研究,分析了顶升和平移速率及地面平整度对结构位移响应的影响,评估了结构的受力和安全状态。研究可为城市更新中历史建筑的整体移位技术提供参考。

齐鲁卢氏内科流派益气温阳法治疗病态窦房结综合征经验

病态窦房结综合征是临床常见难治病。齐鲁卢氏内科流派着眼于“虚-寒-瘀-痰”,认为该病病机以心气心阳虚损为本,寒凝、血瘀、痰阻为标,治以益气温阳、祛瘀化痰通络,疗效显著。附益心复脉汤及验案1则。

通道内模亚克力施工技术与质量控制分析

文章以濮阳至卫辉高速公路濮阳段项目为例,该项目是河南省高速公路网络的关键组成部分,涉及通道底板与侧墙钢筋绑扎工艺。通过详细介绍施工过程,从特殊路基处理、基础开挖、土工格栅铺设、垫层施工、基础混凝土浇筑到底板钢筋控制线弹设、侧墙钢筋卡具使用及亚克力模板等方面,阐述了施工的工艺和质量控制措施,特别强调了钢筋绑扎的重要性,以确保通道的施工质量。通过综合运用这些技术和控制方法,该项目成功实施,并于2020年被评为“2020年河南省样板工程”,为类似工程提供了宝贵经验。

基于盐水冻结加固的地铁隧道联络通道施工技术研究

联络通道作为地铁隧道常用的附属设施,相关的施工技术总结相对欠缺。基于实际工程,针对某中~强风化泥质粉砂岩中地铁隧道联络通道施工进行分析总结。梳理了冻结加固施工过程中的冻结关键技术,明确了冻结施工参数,总结了联络通道施工全过程的关键技术要点,整理了冻结加固地层的联络通道注浆工艺,阐述了注浆加固关键技术,提出了联络通道施工过程中的风险处置措施。

暗挖通道下穿立交桥地层加固施工关键技术及监测分析

地铁车站暗挖通道下穿立交桥的复杂施工工况涉及地下结构建设和地面交通结构的相互协调,高效可靠的地层加固技术及关键指标监测是保障施工顺利进行的重要手段。以济南地铁4号线车站暗挖通道下穿立交桥为例,分析了工程的主要风险,提出超前深孔注浆地层加固技术,并对施工过程中的地表沉降、通道结构变形及立交桥沉降等监测数据进行预警判断分析,并提出相应措施。