基于深度强化学习算法的“电网脑”及其示范工程应用

Deep reinforcement learning-based grid mind and field demonstration application

可再生能源、电力电子设备渗透率持续增大以及大功率交直流混联,电网的动态性、随机性和不确定性显著增强,给电力系统安全稳定运行带来新的挑战。为更有效解决电网中出现的电压、潮流快速波动而导致的安全问题,提出一种基于最大熵深度强化学习算法的智能电网调控辅助决策方法,同时考虑多种控制目标,对电网运行方式进行在线优化控制。该方法将电网调度控制决策建模为马尔科夫决策过程,训练多线程智能体,并采用周期性在线训练机制对智能体的控制性能进行不断提升。基于该方法所研发的辅助决策原型软件部署在国网江苏电力调度控制中心,可与电网调度控制系统环境直接交互,自主学习且不断提升智能体调控决策能力。训练好的智能体可针对电压越限、联络线潮流越限、网损等综合控制目标在毫秒级时间内给出有效控制策略。

With the increasing penetration of renewable energy and power electronics-based devices,and the hybrid operation of AC/DC power networks with heavy power transfer,the dynamics,stochastics and uncertainties of the power grid are being observed,threatening its secure operation.In order to effectively resolve security issues caused by fast variations of voltage and line flows,a reinforcement learning algorithm based on maximum entropy depth is presented for providing online decision support in smart grid operation,which can simultaneously consider multiple control objectives.This method formulates decision derivation for grid operation as Markov decision process,which trains multi-threaded soft actor-critic and uses periodic online training mechanism to continuously improve its control performance.The developed prototype using this method has been deployed in the control center of SGCC Jiangsu electric power company,which interacts with live energy management system and learns its control policy adaptively.The welltrained agent can provide effective control actions within milliseconds to regulate voltage violation,line flow and losses.