基于强化学习的1型糖尿病胰岛素给药策略研究

Study on insulin administration strategy of type 1 diabetes based on reinforcement learning

1型糖尿病(T1D)患者需要通过外源性胰岛素的输送将血糖(BG)维持在治疗范围内。目前,已有的几种基于模型预测控制和强化学习(RL)的胰岛素给药算法存在样本效率差、奖励机制过于简单、血糖调控效果不佳等问题。为此提出了一种基于强化学习的带有指导网络的胰岛素给药策略(insulin administration strategy with guided network,IASGN),针对给药策略安全性能和快速性的特点,引入累积情节奖励和分类经验回放方法,按照不同的重要性采样权重增加了精英样本池,并基于精英样本池训练给药指导网络,对策略网络进行动作指导,改进了奖励机制,在FDA批准的UVA/Padova T1D模拟器中验证了该方法的性能。结果显示,该方法TIR(time in range)达到了98.21%,TBR(time below range)接近于0,CVGA中所有患者均处于A+B区的安全范围,可以使患者血糖长期处于正常范围内,避免了低血糖的风险,在与基准方法对比中也获得了更好的表现。

Type 1 diabetes(T1D)patients need to maintain blood glucose(BG)within the treatment range through the delivery of exogenous insulin.At present,several existing insulin administration algorithms based on model predictive control and reinforcement learning(RL)have problems such as poor sample efficiency,overly simple reward mechanisms,and poor blood glucose regulation effects.This paper proposed an IASGN strategy based on reinforcement learning.Aiming at the characteristics of safety and rapidity of the administration strategy,it introduced cumulative plot rewards and classified experience playback me-thods,increased elite sample pool according to different importance sampling weights,trained the administration guidance network based on the elite sample pool to guide the action of the strategy network,and improved the reward mechanism.It verified the performance of the proposed method in the FDA approved UVA/Padova T1D simulator.The results show that the TIR of the proposed method reaches 98.21%,and the TBR is close to 0.All patients in CVGA are within the safe range of A+B zone,which can keep their blood sugar within the normal range for a long time and avoid the risk of hypoglycemia.Compared with the benchmark methods,it also achieved better performance.