基于历史调控经验的强化学习油藏生产优化方法

Reinforcement Learning Reservoir Production Optimization Method Based on Historical Control Experience

生产优化是调整水驱油流动方向、提高油田开发效果的关键技术。针对现有生产优化方法难以借鉴历史调控经验,优化效率较低的问题。提出基于历史调控经验的强化学习油藏生产优化方法。该方法基于柔性行动器-评判器算法,首先将生产优化建模为马尔可夫序列决策过程,在序列决策过程中,将油藏压力场和饱和度场作为强化学习智能体的观测状态,以开发方案的经济净现值作为智能体的奖励,然后强化学习智能体将油藏状态映射为生产调控制度,并与油藏环境持续交互以累积历史调控经验,最后通过借鉴历史调控经验,快速学习最优策略。将提出的方法应用到油藏区块上进行测试。实验结果表明,本文所提出的基于历史调控经验的强化学习油藏生产优化方法在优化性能和增油控水方面优于现有的进化算法和代理模型方法。

Production optimization is seen as a key technology for adjusting the flow direction of water-driven oil and improving the development effect of oil fields.However,existing optimization methods are often found to struggle in drawing from historical control experience,resulting in suboptimal efficiency.A reinforcement learning reservoir production optimization method based on historical control experience was proposed.The method,based on the soft actor-critic algorithm,was first modeled as a Markov decision process for production optimization.Within this decision process,the pressure and saturation fields of the reservoir were taken as observable states for the reinforcement learning agent,while the economic net present value of development schemes was treated as rewards.Subsequently,reservoir states were mapped to production control schemes by the reinforcement learning agent,which continually interacted with the reservoir environment to accumulate historical control experience.Ultimately,by leveraging this experience,optimal policies were rapidly learned by the agent.The proposed method is applied to a reservoir block for testing.The experimental results show that the proposed reinforcement learning reservoir production optimization method based on historical control experience outperforms traditional evolutionary algorithms and surrogate model methods in terms of optimization performance and oil increase and water control.