星地融合网络中基于深度强化学习的多业务缓存策略

Multi-type Services Caching Strategy Based on Deep Reinforcement Learning in Integrated Satellite-Terrestrial Networks

星地融合网络承载的通信服务呈现出多类型业务并发、业务需求差异化、数据流量聚集、大量重复请求等鲜明特征。针对多样化重复请求业务并发时链路负载过大、用户体验质量(Quality of Experience,QoE)难以保障的问题,提出一种基于深度强化学习(Deep Reinforcement Learning,DRL)的多业务缓存(Caching for Multi-Type Services,CMTS)策略。通过对星地融合网络中获取请求内容时延与三类典型业务时间效用函数分析建模,建立以最大化系统和效用为目标的优化问题,并提出一种基于多智能体深度确定性策略梯度(Multi-Agent Deep Deterministic Policy Gradient,MADDPG)的MADDPG-CMTS算法,综合考虑业务效用差异化特征、用户请求、星地缓存、网络拓扑等多种因素确定缓存更新决策。仿真结果表明,所提算法与最受欢迎内容(Most Popular Content,MPC)策略、随机替换(Random Replacement,RR)策略等传统缓存更新策略相比,系统总效用可提升约47%。

The communication services carried by integrated satellite-terrestrial networks exhibit distinctive characteristics,including multi-type services concurrent,differentiated service requirements,data traffic aggregation,and a large volume of repetitive requests.In response to the issue of excessive link load and difficulty in ensuring the Quality of Experience(QoE) when dealing with diverse repetitive requests,a Caching for Multi-Type Services(CMTS) strategy is obtained based on Deep Reinforcement Learning(DRL).Through analyzing and modeling the delay of obtaining request content and the time utility functions of three typical services in satellite-terrestrial integrated networks,an optimization problem to maximize the total system utility is formulated,and then a MADDPG-CMTS algorithm based on Multi-Agent Deep Deterministic Policy Gradient(MADDPG) is proposed.The strategy comprehensively considers multiple factors such as the differentiation of utility,user request status,satellite-terrestrial caching status,and network topology,to determine cache update decisions.Experimental results demonstrate that the proposed algorithm can increase the total system utility by approximately 47%,when compared with traditional cache update strategies such as Most Popular Content(MPC) strategy and Random Replacement(RR) strategy.