基于PPO算法的CIES低碳优化调度方法

阶梯式碳交易机制以及优化调度模型求解算法是进行园区综合能源系统(community integrated energy system,CIES)优化调度的重要因素,现有文献对这两个因素的考虑不够全面。为此,文中在考虑阶梯式碳交易机制的基础上,提出采用近端策略优化(proximal policy optimization,PPO)算法求解CIES低碳优化调度问题。该方法基于低碳优化调度模型搭建强化学习交互环境,利用设备状态参数及运行参数定义智能体的状态、动作空间及奖励函数,再通过离线训练获取可生成最优策略的智能体。算例分析结果表明,采用PPO算法得到的CIES低碳优化调度方法能够充分发挥阶梯式碳交易机制减少碳排放量和提高能源利用率方面的优势。

基于PPO算法的机械臂抓取策略研究

以解决机械臂在面临多目标场景中执行抓取任务时的路径规划问题为目标,验证PPO算法在机械臂抓取策略上的可行性。研究采用基于PPO算法的深度强化学习,通过在PyBullet仿真环境中进行机械臂与物体的交互模拟,以机械臂末端的位置坐标和目标物体的位置坐标作为状态空间输入到深度强化学习网络,并输出下一步骤的机械臂末端位置坐标,再通过机械臂逆运动学求得各个关节的旋转角度作为动作的输出,最后结合经过优化的奖励函数,提高机械臂在训练抓取任务时的学习效率,加快收敛速度。仿真实验表明,经过优化的奖励函数和PPO算法的应用使得机械臂训练时获取的奖励值能够在第1000回合左右开始收敛到-50~0,且能够在10步左右的步数稳定完成抓取任务,验证了该方法的可行性。该研究成果在机械臂抓取任务中取得很好的效果,为实际应用中解决复杂抓取问题提供了有力的方法和技术支持。

基于PPO的自适应PID控制算法研究

采用MATLAB物理引擎联合Python搭建了一个六轴机械臂,并模拟带有扰动的复杂控制环境,为机械臂训练提供现实中无法提供的试错环境。使用强化学习中近端优化算法(proximal policy optimization,PPO)算法对传统PID控制算法进行改进,引入多智能体思想,根据PID三个参数对控制系统的不同影响及六轴机械臂的特性,将三个参数分别作为不同的智能个体进行训练,实现多智能体自适应调整参数的新型多智能体自适应PID算法。仿真结果表明:该算法的训练收敛性优于MA-DDPG与MA-SAC算法,与传统PID算法的控制效果相比,在遇到扰动及振荡的情况下,能够更有效地抑制振荡,并具有更低的超调量和调整时间,控制过程更为平缓,有效提高了机械臂的控制精度,证明了该算法的鲁棒性及有效性。

Cloning and Sequence Analysis of Prophenoloxidase from Haemocytes of the Red Swamp Crayfish,Procambarus clarkii

The full-length cDNA sequence of prophenoloxidase was obtained through RACE technology. The complete cDNA sequence is 3 721-bp long, containing an open reading frame （ORF） of 1 881 bp, a 154-bp 5′-untranslated region, and a 1 686- bp 3′-untranslated region with three potential functional poly（A） signals （AATAAA）. The molecular mass of the deduced amino acid sequence （627 aa） was 72.3 kDa with an estimatedpI of 5.88. It contained putative copper-binding sites （copper A： 131, 135, 167 and copper B： 301,305, 341）, and a tentative complement-like motif （GCGWPDHL）. Eight potential N-linked glycosylation sites were predicted to be present in P. clarkii prophenoloxidase. Similar to those in other arthropod prophenoloxidases reported so far, no signal peptide was detected in the crayfish prophenoloxidase. The phylogenetic trees confirmed that P. clarkii prophenoloxidase was most closely related to that of freshwater crayfish P. leniusculus and more closely related to other crustacean prophenoloxidases from shrimp, prawn, and lobster than to the insect prophenoloxidases. Besides, two putative introns were found in this sequence of genomic DNA.

Modulation by Biogenic Amines for the Hemocyte Count and Prophenoloxidase Exocytosis via Receptors in Litopenaeus vannamei

Hemocyte counts and phenoloxidase(PO)activity were examined after hemolymph being incubated in dopamine(DA),noradrenaline(NE)and serotonin(5-HT).Results showed that all the three biogenic amines(BAs)had a significant impact on total hemocyte count(THC),differential hemocyte count(DHC),and intracelluar and extracelluar phenoloxidase(PO)activity.Among these Bas,DA had the strongest effect on the above parameters,whereas 5-HT had the least effect.Preincubation with D1 receptor antagonist SCH23390,D2 receptor antagonist Sulpiride and 1:1 admixture of the two could significantly inhibit the effect of DA on these parameters.SCH23390 showed a stronger inhibitory effect than Sulpiride,and the admixture exhibited the strongest effect.These results suggested that the change of hemocyte count and activation of prophenoloxidase(proPO)system in Litopenaeusvan-namei hemocyte can be regulated by BAs,and DA modulates the two parameters via its receptors.

Effect of Dopamine Injection on the Hemocyte Count and Prophenoloxidase System of the White Shrimp Litopenaeus vannamei

Effects of dopamine injection on the hemocyte count, phenoloxidase activity, serine proteinase activity, proteinase in-hibitor activity and α2-macroglobulin-like activity in L. vannamei were studied. Results showed that dopamine injection resulted in a significant effect on the parameters measured (P < 0.05), while no significant difference was observed in the control group (0.85% NaCl). In the experimental groups,the hemocyte count reached the minimum in 3 h;granular and semi-granular cells became stable after 12 h and hyaline cells and the total hemocyte count became stable after 18 h. Phenoloxidase activity reached the minimum in 6 h, and then became stable after 9 h. Serine protease activity and proteinase inhibitor activity reached the minimum in 3 h, and α2-macro-globulin-like activity reached the maximum in 3 h,and all the three parameters became stable after 12 h. The results suggest that the activating mechanisms of the proPO system triggered by dopamine are different from those triggered by invasive agents or sponta-neously activated under a normal physical condition.

基于样本优化的PPO算法在单路口信号控制的应用

优化交通信号的控制策略可以提高道路车辆通行效率,缓解交通拥堵.针对基于值函数的深度强化学习算法难以高效优化单路口信号控制策略的问题,构建了一种基于样本优化的近端策略优化(MPPO)算法的单路口信号控制方法,通过对传统PPO算法中代理目标函数进行最大化提取,有效提高了模型选择样本的质量,采用多维交通状态向量作为模型观测值的输入方法,以及时跟踪并利用道路交通状态的动态变化过程.为了验证MPPO算法模型的准确性和有效性,在城市交通微观模拟软件(SUMO)上与值函数强化学习控制方法进行对比.仿真实验表明,相比于值函数强化学习控制方法,该方法更贴近真实的交通场景,显著加快了车辆累计等待时间的收敛速度,车辆的平均队列长度和平均等待时间明显缩短,有效提高了单路口车辆的通行效率.

Responses of Prophenoloxidase System and Related Defence Parameters of Litopenaeus vannamei to Low Salinity

In this study,we investigated the effects of low salinity(26 and 21) on the prophenoloxidase(proPO) system and related defence parameters in the shrimp Litopenaeus vannamei.The results showed that low salinity induced a significant increase of dopamine(DA) concentration in haemolymph at 6 h of the experiment;on the other hand,total haemocyte count(THC),differential haemocyte count(DHC) and PO activity decreased over time to the lowest level at 24 h and remained low thereafter.Serine Protease(SP) and Proteinase Inhibitor(PI) activity in the two lower salinity treatments decreased to the lowest level at 12 and 24 h,respectively,and both recovered to the control level at 72 h.In contrast,α2-macroglobulin(α2M) activity in the two lower salinity treatments peaked at 24 h and then decreased to the control level at 72 h.Therefore,it may be concluded that stress-induced DA plays an important temporary role in neurotransmission and causes immune response in L.vannamei in adapting to salinity changes.

Effects of acute salinity stress on the survival and prophenoloxidase system of Exopalaemon carinicauda

The ridgetail white prawn Exopalaemon carinicauda is a euryhaline shrimp species in the estuarine and coastal areas of China.In this study,survival rates,transcription levels of two prophenoloxidase system-related genes(Ec LGBP and Ecpro PO)and PO activity were determined quantitatively in juvenile and adult E.carinicauda under different salinity levels.The results showed that E.carinicauda juveniles could survive in a wider range of salinity conditions than adults.For juvenile E.carinicauda,the expression levels of Ec LGBP and Ec Pro PO were upregulated in low salinities and showed no significant difference at 20–40,while PO activities in low salinities were higher compared to those in high salinities.For adult E.carinicauda,the expression profiles of Ec LGBP and Ecpro PO had a different trend of up-regulation in salinity stress treatments and no obvious difference was observed in the gene expression levels and PO activity between 30 and 40.The salinity tolerance range of immunity for juvenile and adult E.carinicauda is 20–40 and 30–40,respectively.

基于一种改进PPO算法的无人机空战自主机动决策方法研究

深度强化学习的应用为无人机自主机动决策提供了新的可能。提出一种基于态势评估模型重构与近端策略优化(PPO)算法相结合的无人机自主空战机动决策方法,为一对一近距空战提供了有效策略选择。首先,建立高保真六自由度无人机模型与近距空战攻击模型;其次,基于空战状态划分重构角度、速度、距离和高度态势函数,提出一种描述机动潜力的新型态势评估指标;之后,基于态势函数设计塑形奖励,并与基于规则的稀疏奖励、基于状态转换的子目标奖励共同构成算法奖励函数,增强了强化学习算法的引导能力;最后,设计专家系统作为对手,在高保真空战仿真平台(JSBSim)中对本文工作进行了评估。仿真验证,应用本文方法的智能体在对抗固定机动对手与专家系统对手时算法收敛速度与胜率都得到了有效提升。

基于PPO的球形机器人目标跟随研究

球形机器人由于其优异的运动性能、出色的地形适应能力和防侧翻的特性,被广泛应用于水下探测、岸滩巡检等需要适应复杂环境的场景。然而球形机器人系统模型具有欠驱动、非线性的特点,运动控制问题复杂,在复杂应用环境下难以可靠跟随目标。为此,提出了一种基于近端策略优化(PPO)算法的球形机器人目标跟随方法。该方法基于深度强化学习理论,在球形机器人动力学模型的基础上,设计了简单高效的动作空间和表征完善的状态空间。并且为提高目标跟随方法的鲁棒性,该方法在奖励函数中引入人工势场,以使目标始终保持在机器人视野中心。仿真结果表明,所提方法能够满足既定场景的跟随需求,球形机器人使用该方法可以对随机运动目标进行可靠跟随。

PPO增韧生物基PA56合金的制备及性能

以有机刚性粒子PPO为增韧剂,采用熔融共混的方法对PA56进行增韧改性,制得PA56/PPO合金。研究结果表明,PA56/PPO合金样品断面形貌呈现典型的海-岛结构,PPO以球状颗粒的形态均匀地分布在PA56连续相中,相畴尺寸为100~300 nm。PA56/PPO合金的力学性能测试表明,随着PPO含量的增加,简支梁缺口冲击强度增强,断裂伸长率提高,当PPO质量分数为30%时,PA56/PPO合金的冲击强度为9.4 kJ/m^(2),断裂伸长率为20.5%,与纯PA56相比,分别提高了303%和820%;但是,合金的拉伸强度变化较小。当PPO质量分数为30%,PA56/PPO合金吸水率为1.67%,与纯PA56相比降低了57.6%。随着体系中PPO含量的增加,合金材料的结晶温度和结晶度逐渐降低,热失重并未发生明显改变。

基于PPO算法的不同驾驶风格跟车模型研究

自动驾驶对于减少交通堵塞、提高驾驶舒适性具有非常重要的作用,如何提高人们对自动驾驶技术的接受程度仍具有重要的研究意义。针对不同需求的人群定制不同的驾驶风格,可以帮助驾驶人理解自动驾驶行为,提高驾驶人的乘车体验,在一定程度上消除驾驶人对使用自动驾驶系统的心理抵抗性。通过分析自动驾驶场景下的跟车行为,提出基于PPO算法的不同驾驶风格的深度强化学习模型设计方案。首先分析德国高速公路车辆行驶数据集(HDD)中大量驾驶行为轨迹,根据跟车时距(THW)、跟车距离(DHW)、行车加速度以及跟车速度特征进行归类,提取激进型的驾驶风格和稳健型的驾驶风格的特征数据,以此为基础编码能够反映驾驶人风格的奖励函数,经过迭代学习生成不同驾驶风格的深度强化学习模型,并在highway env平台上进行道路模拟。实验结果表明,基于PPO算法的不同风格驾驶模型具有完成任务目标的能力,且与传统的智能驾驶模型(IDM)相比,能够在驾驶行为中准确反映出不同的驾驶风格。

PPOS方案及拮抗剂方案在卵巢储备功能减退患者中的应用比较

目的 对比高孕激素状态下促排卵(PPOS)方案及拮抗剂方案在卵巢储备功能减退(DOR)患者中的促排卵效果、实验室培养及首次冻融胚胎移植(FET)妊娠结局,为改善DOR患者的辅助生殖助孕结局提供科学参考。方法 回顾性分析2016年1月—2022年6月本院生殖医学中心收治的302周期DOR患者的临床资料,根据促排卵方案分为PPOS组(216周期)和拮抗剂组(86周期),根据年龄将PPOS组分为:(1)低龄组:78周期,年龄≤35岁;(2)高龄组:138周期,年龄>35岁。同理,将拮抗剂组分为:(1)低龄组:36周期,年龄≤35岁;(2)高龄组:50周期,年龄>35岁。比较各组患者的临床资料、促排卵效果、实验室培养及首次FET妊娠结局。结果 两组患者的年龄、不孕年限、身体质量指数(BMI)、不孕类型、受精方式、基础FSH及LH水平等资料比较均无差异(P>0.05)。以35岁作为划分生育年龄节点,不同促排卵方案的低龄组及高龄组患者的基础FSH及LH水平具有差异性(P<0.05)。PPOS组患者与拮抗剂组患者的Gn启动量、Gn天数、Gn总剂量、卵子逃逸率、未取出卵子率、无可用卵子率、周期取消率等指标无差异(P>0.05);低龄及高龄DOR患者的获卵数相比,PPOS组均低于拮抗剂组,差异具有统计学意义(P<0.05)。低龄及高龄DOR患者中,两种促排卵方案的可用胚胎率、优质胚胎率、2PN率、2PN卵裂率比较均无差异(P>0.05),拮抗剂组卵子利用率更高,其MⅡ数高于PPOS组,差异具有统计学意义(P<0.05)。统计首次FET妊娠结局,PPOS组低龄患者妊娠率和活产率分别为50.00%、27.27%,高于高龄患者的43.33%、12.73%;拮抗剂组低龄患者妊娠率和活产率均为60.00%,高于高龄患者的35.71%、21.43%,但两种促排卵方案同年龄段患者的妊娠率、活产率比较差异无统计学意义(P>0.05)。结论 PPOS方案促排卵方法简单、费用低,已较大规模的成功助孕DOR患者,但在高龄DOR患者中易发生卵子早排。拮抗剂方案卵子利用度高,优质胚胎率、可用胚胎率及临床妊娠结局与PPOS方案相当,且可以新鲜移植,缩短了患者周期内妊娠时间,近年来更多地应用于DOR患者,但启动周期总费用较高。在临床工作中,应根据DOR患者具体情况,严格掌握不同促排卵方案的适应证与禁忌症,制定安全、有效、个体化的促排卵方案,以获得良好的妊娠结局。

Channel assignment and power allocation for throughput improvement with PPO in B5G heterogeneous edge networks

In Beyond the Fifth Generation(B5G)heterogeneous edge networks,numerous users are multiplexed on a channel or served on the same frequency resource block,in which case the transmitter applies coding and the receiver uses interference cancellation.Unfortunately,uncoordinated radio resource allocation can reduce system throughput and lead to user inequity,for this reason,in this paper,channel allocation and power allocation problems are formulated to maximize the system sum rate and minimum user achievable rate.Since the construction model is non-convex and the response variables are high-dimensional,a distributed Deep Reinforcement Learning(DRL)framework called distributed Proximal Policy Optimization(PPO)is proposed to allocate or assign resources.Specifically,several simulated agents are trained in a heterogeneous environment to find robust behaviors that perform well in channel assignment and power allocation.Moreover,agents in the collection stage slow down,which hinders the learning of other agents.Therefore,a preemption strategy is further proposed in this paper to optimize the distributed PPO,form DP-PPO and successfully mitigate the straggler problem.The experimental results show that our mechanism named DP-PPO improves the performance over other DRL methods.

基于强化学习PPO算法的上市公司投资组合管理

传统的投资组合管理方法往往依赖于经验规则或数学模型,难以充分利用市场信息和动态调整投资策略。为了解决这一问题,文章提出一种基于强化学习PPO(Proximal Policy Optimization)算法的新方法。使用上市公司的历史数据进行训练和测试,与传统投资策略和其他强化学习算法进行比较,实验结果表明,基于强化学习PPO算法的投资组合管理方法在投资回报率和风险控制方面取得了显著的改进。

PPOS方案、LPOS方案和拮抗剂方案在POR患者IVF/ICSI-ET中的临床应用效果

分析卵泡期外源性高孕激素下促排卵方案(progestin-primed ovarian stimulation,PPOS),黄体期内源性高孕激素下促排卵(luteal-phase ovarian stimulation,LPOS)方案和拮抗剂方案在高龄卵巢低反应((poor ovarian response,POR)患者体外受精/卵胞质内单精子注射-胚胎移植(in vitro fertilization/intra cytoplasmicsperm injection-embryo transfer,IVF/ICSI-ET)助孕中的临床应用效果。方法 选取本院2023年01月至2023年12月间100例IVF/ICSI-ET助孕POR患者作为观察对象,依据促排卵方案不同,分为3组,对照组(拮抗剂方案)(38例)、观察1组(PPOS方案)(32例)、观察2组(LPOS方案)(30例),比较应用效果。结果 观察1组、观察2组促排卵周期资料无统计学意义(P>0.05),但优于对照组(P<0.05);获卵数、周期取消率、卵裂数等实验室指标,观察1组、观察2组优于对照组(P<0.05)。结论 与拮抗剂相比,LPOS、PPOS方案更好,均可起到显著促排卵效果。

基于PPO算法的光储式充电站储能运行策略

电动汽车的随机充电与光伏出力的波动特性给光储式充电站的经济高效运行提出了新的要求。为此,提出了一种基于深度强化学习近端策略优化(Proximal Policy Optimization,PPO)算法的光储式充电站储能运行策略。首先,对光储式充电站系统结构进行了分析,并建立了以综合运行成本最小为目标的储能运行模型。其次,将该模型转化为马尔科夫决策过程,并采用深度强化学习PPO算法进行训练求解。最后,在一个光储式充电站中进行算例仿真,验证所提策略的有效性。

高温强光胁迫对苹果果皮PPO活性的影响

苹果果实日烧是一种普遍发生的生理病害,最常见的特征之一就是在果实表面出现褐变。通常认为PPO与植物的酶促褐变密切相关。研究了自然和控制条件下,高温强光胁迫对果实PPO活性的影响,以便揭示高温强光胁迫下苹果果实褐变与PPO活性之间的联系。结果表明:高温和强光胁迫与果皮PPO活性变化密切相关。就树冠不同方位而言,西南方位是高温和强光胁迫最严重的区域,其外围裸露果实的PPO活性也最强。在一定范围内,随着处理温度和光照强度的升高,果皮PPO活性也逐渐增强。短时间剧烈升温能够引起PPO活性骤然上升。在同样程度的高温胁迫下,提高环境湿度有利于抑制果皮PPO活性,从而减轻褐变症状的发生。室内外试验一致证实:果实日烧褐变现象与高温强光胁迫下果皮组织PPO活性大幅度提高有直接关系。

放线菌剂对黄瓜幼苗生长及叶片PPO活性的影响

以黄瓜幼苗为试材,采用盆栽试验研究了3种生防放线菌剂单接和与病原菌混接对黄瓜的促生作用及对黄瓜叶片PPO活性的影响。结果表明:①供试放线菌制剂对黄瓜幼苗有显著的促生作用,在接种量为1.5 g/kg、20 d时,菌剂Act12(密旋链霉菌,Streptomyces pactum)单独接种处理后,黄瓜植株总质量、根系鲜质量、根系长度、根系活力及叶面积分别较对照CK提高18.8%、225.0%、57.3%、482.0%及45.3%;Act12与黄瓜枯萎菌HK混合接种时,其各对应指标测值分别较对照HK提高78.9%、114.6%、162.6%、443.4%及50.0%。②施用菌剂后对叶片诱导酶PPO活性有一定提高,45 d时,单接Act12(1.5 g/kg)较对照增加了54.4%。③接种放线菌剂对黄瓜幼苗叶片绿色度SPDA无明显影响。以上结果表明,施用放线菌剂能显著促进黄瓜生长发育,对根系生长及活力的促进作用尤为明显,并使黄瓜产生诱导抗性。