Combined peakreductionandself-consumptionusingproximalpolicy optimisation

Residential demand response programs aim to activate demand flexibility at the household level.In recent years,reinforcement learning(RL)has gained significant attention for these type of applications.A major challenge of RL algorithms is data efficiency.New RL algorithms,such as proximal policy optimisation(PPO),have tried to increase data efficiency.Addi tionally,combining RL with transfer learning has been proposed in an effort to mitigate this challenge.In this work,we further improve upon state-of-the-art transfer learning performance by incorporating demand response domain knowledge into the learning pipeline.We evaluate our approach on a demand response use case where peak shaving and self-consumption is incentivised by means of a capacity tariff.We show our adapted version of PPO,combined with transfer learming,reduces cost by 14.51%compared to a regular hysteresis controller and by 6.68%compared to traditional PPO.

Direct Load Control of Thermostatically Controlled Loads Based on Sparse Observations Using Deep Reinforcement Learning

This paper considers a demand response agent that must find a near-optimal sequence of decisions based on sparse observations of its environment.Extracting a relevant set of features from these observations is a challenging task and may require substantial domain knowledge.One way to tackle this problem is to store sequences of past observations and actions in the state vector,making it high dimensional,and apply techniques from deep learning.This paper investigates the capabilities of different deep learning techniques,such as convolutional neural networks and recurrent neural networks,to extract relevant features for finding near-optimal policies for a residential heating system and electric water heater that are hindered by sparse observations.Our simulation results indicate that in this specific scenario,feeding sequences of time-series to an Long Short-Term Memory(LSTM)network,which is a specific type of recurrent neural network,achieved a higher performance than stacking these time-series in the input of a convolutional neural network or deep neural network.

Transfer learning in demand response: A review of algorithms for data-efficient modelling and control

A number of decarbonization scenarios for the energy sector are built on simultaneous electrification of energy demand,and decarbonization of electricity generation through renewable energy sources.However,increased electricity demand due to heat and transport electrification and the variability associated with renewables have the potential to disrupt stable electric grid operation.To address these issues using demand response,researchers and practitioners have increasingly turned towards automated decision support tools which utilize machine learning and optimization algorithms.However,when applied naively,these algorithms suffer from high sample complexity,which means that it is often impractical to fit sufficiently complex models because of a lack of observed data.Recent advances have shown that techniques such as transfer learning can address this problem and improve their performance considerably—both in supervised and reinforcement learning contexts.Such formulations allow models to leverage existing domain knowledge and human expertise in addition to sparse observational data.More formally,transfer learning embodies all techniques where one aims to increase(learning)performance in a target domain or task,by using knowledge gained in a source domain or task.This paper provides a detailed overview of state-of-the-art techniques on applying transfer learning in demand response,showing improvements that can exceed 30%in a variety of tasks.We observe that most research to date has focused on transfer learning in the context of electricity demand prediction,although reinforcement learning based controllers have also seen increasing attention.However,a number of limitations remain in these studies,including a lack of benchmarks,systematic performance improvement tracking,and consensus on techniques that can help avoid negative transfer.