Multi-agent Simulation for Strategic Bidding in Electricity Markets Using Reinforcement Learning

In this paper,a theoretical framework of Multiagent Simulation(MAS)is proposed for strategic bidding in electricity markets using reinforcement learning,which consists of two parts:one is a MAS system used to simulate the competitive bidding of the actual electricity market;the other is an adaptive learning strategy bidding system used to provide agents with more intelligent bidding strategies.An ExperienceWeighted Attraction(EWA)reinforcement learning algorithm(RLA)is applied to the MAS model and a new MAS method is presented for strategic bidding in electricity markets using a new Improved EWA(IEWA).From both qualitative and quantitative perspectives,it is compared with three other MAS methods using the Roth-Erev(RE),Q-learning and EWA.The results show that the performance of the MAS method using IEWA is proved to be better than the others.The four MAS models using four RLAs are built for strategic bidding in electricity markets.Through running the four MAS models,the rationality and correctness of the four MAS methods are verified for strategic bidding in electricity markets using reinforcement learning.

Selective Learning for Strategic Bidding in Uniform Pricing Electricity Spot Market

In an electricity market,power producers’profits are determined by the market price instead of the regulated price.Therefore,the producers should be cautious in strategic bidding,for which prediction-based approaches are widely used and have been proved effective for many areas.However,in a uniform pricing market,the market environment is so complicated,which is primarily due to the complexity of the participants’interaction,that even the strategies based on machine learning algorithms,which are generally considered as outstanding nonlinear prediction methods,may sometimes lead to unsatisfactory results.Therefore,a selective learning scheme for strategic bidding is proposed to ensure greater effectiveness.The proposed scheme is based on an ensemble technique,where several machine learning algorithms serve as the underlying algorithms to predict the price and generate a bidding recommendation.As the clearing iteration progresses,the most fitting ones will be chosen to dominate the bidding strategy.Considering the characteristics of the electricity market,the prediction method used in the selective learning scheme is modified to achieve higher accuracy.Simulation studies are presented to demonstrate the effectiveness of the proposed scheme,which leads to more reasonable bidding behaviors and higher profits.

Impact of Long-Term Transactions on Strategic Bidding in the Electricity Market

Power producers'profits are determined by the market price in the electricity market and therefore they will adopt certain strategies in market transactions to achieve higher profits.In an electricity spot market that adopts uniform pricing,power producers with considerable generation capacity are able to exercise their market power,given that the market concentration is relatively high at the beginning of market reform.It has been proved that an effective bidding strategy can increase the market clearing prices,so as to increase the profits of the power producer.Fortunately,the introduction of long-term transactions may mitigate the impact of producers'market power,as a great amount of long-term volume is settled at the long-term contract price,which is determined in advance and is less fluctuating than the spot price.Two forms of long-term transactions,the fixed volume contract and the varied volume contract,are studied in this paper.Simulation studies are conducted on a multi-agent platform,where both the long-term and spot transactions of power producers are included,and the total profits of a power producer with or without long-term transactions are analyzed to demonstrate their influence.Meanwhile,the results clearly show that long-term transactions can effectively prevent power producers from exercising market power.

Market Equilibrium Based on Cloud-edge Collaboration

Market participants can only bid with lagged information disclosure under the existing market mechanism,which can lead to information asymmetry and irrational market behavior,thus influencing market efficiency.To promote rational bidding behavior of market participants and improve market efficiency,a novel electricity market mechanism based on cloudedge collaboration is proposed in this paper.Critical market information,called residual demand curve,is published to market participants in real-time on the cloud side,while participants on the edge side are allowed to adjust their bids according to the information disclosure prior to closure gate.The proposed mechanism can encourage rational bids in an incentive-compatible way through the process of dynamic equilibrium while protecting participants’privacy.This paper further formulates the mathematical model of market equilibrium to simulate the process of each market participant’s strategic bidding behavior towards equilibrium.A case study based on the IEEE 30-bus system shows the proposed market mechanism can effectively guide bidding behavior of market participants,while condensing exchanged information and protecting privacy of participants.

An Efficient Decomposition Method for Bilevel Energy Storage Arbitrage Problem

With the reduction of cost,large-capacity energy storage unit is playing an increasingly important role in modern power systems.When a merchant energy storage unit participates in the power market,its arbitrage problem can be modeled via a bilevel program.The lower-level problem simulates power market clearing and gives the nodal price,based on which the upperlevel problem maximizes the arbitrage profit of energy storage.To solve this bilevel problem,the conventional method replaces the lower level problem with its KKT optimality conditions and further performs linearization.However,because the size of the market clearing problem grows with the scale of the power system and the number of periods,the resulting MILP(mixed-integer linear program)is very challenging to solve.This paper proposes a decomposition method to address the bilevel energy storage arbitrage problem.First,the locational marginal price at the storage connection node is expressed as a piecewise constant function in the storage bidding strategy,so the market clearing problem can be omitted.Then,the storage bidding problem is formulated as a mixed-integer linear program,which contains only a few binary variables.Numeric experiments validate the proposed method is exact and highly efficient.