Managing power grids through topology actions: A comparative study between advanced rule-based and reinforcement learning agents

The operation of electricity grids has become increasingly complex due to the current upheaval and the increase in renewable energy production.As a consequence,active grid management is reaching its limits with conventional approaches.In the context of the Learning to Run a Power Network(L2RPN)challenge,it has been shown that Reinforcement Learning(RL)is an efficient and reliable approach with considerable potential for automatic grid operation.In this article,we analyse the submitted agent from Binbinchen and provide novel strategies to improve the agent,both for the RL and the rule-based approach.The main improvement is a N-1 strategy,where we consider topology actions that keep the grid stable,even if one line is disconnected.More,we also propose a topology reversion to the original grid,which proved to be beneficial.The improvements are tested against reference approaches on the challenge test sets and are able to increase the performance of the rule-based agent by 27%.In direct comparison between rule-based and RL agent we find similar performance.However,the RL agent has a clear computational advantage.We also analyse the behaviour in an exemplary case in more detail to provide additional insights.Here,we observe that through the N-1 strategy,the actions of both the rule-based and the RL agent become more diversified.

A Reinforcement Learning approach for the continuous electricity market ofGermany: Trading from the perspective of a wind park operator

With the rising extension of renewable energies, the intraday electricity markets have recorded a growingpopularity amongst traders as well as electric utilities to cope with the induced volatility of the energysupply. Through their short trading horizon and continuous nature, the intraday markets offer the abilityto adjust trading decisions from the day-ahead market or reduce trading risk in a short-term notice. Producersof renewable energies utilize the intraday market to lower their forecast risk, by modifying their providedcapacities based on current forecasts. However, the market dynamics are complex due to the fact that thepower grids have to remain stable and electricity is only partly storable. Consequently, robust and intelligenttrading strategies are required that are capable to operate in the intraday market. In this work, we proposea novel autonomous trading approach based on Deep Reinforcement Learning (DRL) algorithms as a possiblesolution. For this purpose, we model the intraday trade as a Markov Decision Process (MDP) and employ theProximal Policy Optimization (PPO) algorithm as our DRL approach. A simulation framework is introducedthat enables the trading of the continuous intraday price in a resolution of one minute steps. We test ourframework in a case study from the perspective of a wind park operator. We include next to general tradeinformation both price and wind forecasts. On a test scenario of German intraday trading results from 2018,we are able to outperform multiple baselines with at least 45.24% improvement, showing the advantage of theDRL algorithm. However, we also discuss limitations and enhancements of the DRL agent, in order to increasethe performance in future works.