基于参数共享机制多智能体深度强化学习的社区能量管理协同优化

Parameter Sharing Empowered Multi-agent Deep Reinforcement Learning for Coordinated Management of Energy Communities

智能电网背景下社区和端对端电能交易有助于挖掘利用产消者分布式能源的灵活性并最大化其价值。尽管多智能体深度强化学习提供了合适的无模型框架以实现多个产消者间能量管理策略的协同优化,该方法仍存在环境状态不稳定、产消者隐私保护和计算复杂度高等局限。该文提出一种将参数共享与优先深度确定性策略梯度法相结合的多智能体强化学习方法,通过智能体间的策略与经验共享以提升学习效率,并降低训练难度。接着构建端对端交易平台以协同社区市场内产消者的电能交易;执行奖励修正以避免产生新的负荷/发电高峰,从而保护本地配网的安全运行;作为可信任第三方向产消者提供有关社区市场的全局信息,在保护产消者隐私的同时减轻环境不稳定性,并提升算法的可扩展性。最后,通过算例验证所提方法能够有效降低社区总运行成本,保证产消者的利益,且较现有算法提高了训练速率与可扩展性。

Energy communities and peer-to-peer trading have arisen recently as promising concepts for maximizing the value streams of distributed energy resources of prosumers in the smart grid setting. Although multi-agent deep reinforcement learning(MADRL) constitutes a suitable model-free framework to coordinate energy management decisions of prosumers, its application to the large-scale coordinated management of multiple prosumers interacting within a community is still in its infancy, owning to the non-stationarity, dimensionality, and privacy drawbacks of state-of-the-art MADRL approaches. This paper proposed a novel MADRL method, combining a parameter sharing approach with a prioritized deep deterministic policy gradient algorithm, which allowed sharing of experiences and learned policies between all agents to ease training. This method was complemented by a peer-to-peer trading platform coordinate prosumers’ participation in the community market, penalized the contribution of each prosumer to rebound peaks through a novel reward shaping mechanism, and acted as a trusted third party informing prosumers in the collective trading behavior of the community market. Case studies demonstrate the superiority of the proposed method over the state-of-the-art approaches.