Intelligent Power Grid Load Transferring Based on Safe Action-Correction Reinforcement Learning

When a line failure occurs in a power grid, a load transfer is implemented to reconfigure the network by changingthe states of tie-switches and load demands. Computation speed is one of the major performance indicators inpower grid load transfer, as a fast load transfer model can greatly reduce the economic loss of post-fault powergrids. In this study, a reinforcement learning method is developed based on a deep deterministic policy gradient.The tedious training process of the reinforcement learning model can be conducted offline, so the model showssatisfactory performance in real-time operation, indicating that it is suitable for fast load transfer. Consideringthat the reinforcement learning model performs poorly in satisfying safety constraints, a safe action-correctionframework is proposed to modify the learning model. In the framework, the action of load shedding is correctedaccording to sensitivity analysis results under a small discrete increment so as to match the constraints of line flowlimits. The results of case studies indicate that the proposed method is practical for fast and safe power grid loadtransfer.

Price-Based Residential Demand Response Management in Smart Grids:A Reinforcement Learning-Based Approach

This paper studies price-based residential demand response management(PB-RDRM)in smart grids,in which non-dispatchable and dispatchable loads(including general loads and plug-in electric vehicles(PEVs))are both involved.The PB-RDRM is composed of a bi-level optimization problem,in which the upper-level dynamic retail pricing problem aims to maximize the profit of a utility company(UC)by selecting optimal retail prices(RPs),while the lower-level demand response(DR)problem expects to minimize the comprehensive cost of loads by coordinating their energy consumption behavior.The challenges here are mainly two-fold:1)the uncertainty of energy consumption and RPs;2)the flexible PEVs’temporally coupled constraints,which make it impossible to directly develop a model-based optimization algorithm to solve the PB-RDRM.To address these challenges,we first model the dynamic retail pricing problem as a Markovian decision process(MDP),and then employ a model-free reinforcement learning(RL)algorithm to learn the optimal dynamic RPs of UC according to the loads’responses.Our proposed RL-based DR algorithm is benchmarked against two model-based optimization approaches(i.e.,distributed dual decomposition-based(DDB)method and distributed primal-dual interior(PDI)-based method),which require exact load and electricity price models.The comparison results show that,compared with the benchmark solutions,our proposed algorithm can not only adaptively decide the RPs through on-line learning processes,but also achieve larger social welfare within an unknown electricity market environment.

Uplift of Symmetrical Anchor Plates by Using Grid-Fixed Reinforced Reinforcement in Cohesionless Soil

Uplift response of symmetrical anchor plates with and without grid fixed reinforced （GFR） reinforcement was evaluated in model tests and numerical simulations by Plaxis. Many variations of reinforcement layers were used to reinforce the sandy soil over symmetrical anchor plates. In the current research, different factors such as relative density of sand, embedment ratios, and various GFR parameters including size, number of layers, and the proximity of the layer to the symmetrical anchor plate were investigated in a scale model. The failure mechanism and the associated rupture surface were observed and evaluated. GFR, a tied up system made of fiber reinforcement polymer （FRP） strips and end balls, was connected to the geosynthetic material and anchored into the soil. Test results showed that using GFR reinforcement significantly improved the uplift capacity of anchor plates. It was found that the inclusion of one layer of GFR, which rested directly on the top of the anchor plate, was more effective in enhancing the anchor capacity itself than other methods. It was found that by including GFR the uplift response was improved by 29%. Multi layers of GFR proved more effective in enhancing the uplift capacity than a single GFR reinforcement. This is due to the additional anchorage provided by the GFR at each level of reinforcement. In general, the results show that the uplift capacity of symmetrical anchor plates in loose and dense sand can be significantly increased by the inclusion of GFR. It was also observed that the inclusion of GFR reduced the requirement for a large L/D ratio to achieve the required uplift capacity. The laboratory and numerical analysis results are found to be in agreement in terms of breakout factor and failure mechanism pattern.

Reinforcement Learning-Based Control for Resilient Community Microgrid Applications

A novel microgrid control strategy is presented in this paper. A resilient community microgrid model, which is equipped with solar PV generation and electric vehicles (EVs) and an improved inverter control system, is considered. To fully exploit the capability of the community microgrid to operate in either grid-connected mode or islanded mode, as well as to achieve improved stability of the microgrid system, universal droop control, virtual inertia control, and a reinforcement learning-based control mechanism are combined in a cohesive manner, in which adaptive control parameters are determined online to tune the influence of the controllers. The microgrid model and control mechanisms are implemented in MATLAB/Simulink and set up in real-time simulation to test the feasibility and effectiveness of the proposed model. Experiment results reveal the effectiveness of regulating the controller’s frequency and voltage for various operating conditions and scenarios of a microgrid.

Deep Reinforcement Learning Based Resource Allocation for Fault Detection with Cloud Edge Collaboration in Smart Grid

Real-time fault detection is important for operation of smart grid.It has become a trend of future development to design an anomaly detection system based on deep learning by using the powerful computing power of the cloud.However,delay of Internet transmission is large,which may make the delay time of detection and transmission go beyond the limits.However,the edge-based scheme may not be able to undertake all data detection tasks due to limited computing resources of edge devices.Therefore,we propose a cloud-edge collaborative smart grid fault detection system,next to which edge devices are placed,and equipped with a lightweight neural network with different precision for fault detection.In addition,a sub-optimal and realtime communication and computing resource allocation method is proposed based on deep reinforcement learning.This method greatly speeds up solution time,which can meet the requirements of data transmission delay,maximize the system throughput,and improve communication efficiency.Simulation results show the scheme is superior in transmission delay and improves real-time performance of the smart grid detection system.

Distributional Reinforcement Learning with Quantum Neural Networks

Traditional reinforcement learning (RL) uses the return, also known as the expected value of cumulative random rewards, for training an agent to learn an optimal policy. However, recent research indicates that learning the distribution over returns has distinct advantages over learning their expected value as seen in different RL tasks. The shift from using the expectation of returns in traditional RL to the distribution over returns in distributional RL has provided new insights into the dynamics of RL. This paper builds on our recent work investigating the quantum approach towards RL. Our work implements the quantile regression (QR) distributional Q learning with a quantum neural network. This quantum network is evaluated in a grid world environment with a different number of quantiles, illustrating its detailed influence on the learning of the algorithm. It is also compared to the standard quantum Q learning in a Markov Decision Process (MDP) chain, which demonstrates that the quantum QR distributional Q learning can explore the environment more efficiently than the standard quantum Q learning. Efficient exploration and balancing of exploitation and exploration are major challenges in RL. Previous work has shown that more informative actions can be taken with a distributional perspective. Our findings suggest another cause for its success: the enhanced performance of distributional RL can be partially attributed to its superior ability to efficiently explore the environment.

Reinforcement Learning with Deep Quantum Neural Networks

The advantage of quantum computers over classical computers fuels the recent trend of developing machine learning algorithms on quantum computers, which can potentially lead to breakthroughs and new learning models in this area. The aim of our study is to explore deep quantum reinforcement learning (RL) on photonic quantum computers, which can process information stored in the quantum states of light. These quantum computers can naturally represent continuous variables, making them an ideal platform to create quantum versions of neural networks. Using quantum photonic circuits, we implement Q learning and actor-critic algorithms with multilayer quantum neural networks and test them in the grid world environment. Our experiments show that 1) these quantum algorithms can solve the RL problem and 2) compared to one layer, using three layer quantum networks improves the learning of both algorithms in terms of rewards collected. In summary, our findings suggest that having more layers in deep quantum RL can enhance the learning outcome.

FRP网格-UHPC复合加固RC梁抗弯性能有限元分析

我国建筑、桥梁等结构大多已接近使用年限,亟需加固改造。为解决新旧接触界面易产生剥离、新加结构与原有部分无法共同受力的问题,改善传统加固方法,采用FRP网格-UHPC复合加固钢筋混凝土梁。用ABAQUS有限元软件进行数值模拟分析,模拟结果与试验结果吻合较好,能够准确模拟FRP网格增强超高性能混凝土(UHPC)复合加固钢筋混凝土梁的抗弯性能。分析FRP网格横、纵向结点数和复合加固层粘结长度等参数对加固梁受弯性能的影响,结果显示,随着粘结长度的增加,加固效果提升显著,承载力最大提升幅度为121.929%;通过增加横向网格结点数,可以有效地增强FRP网格与UHPC之间的机械锚固能力,从而使承载力提升。研究成果为FRP网格-UHPC加固混凝土结构提供参考。

基于强化学习的含电动汽车虚拟电厂优化调度

大量电动汽车(EV)用户的无序充电可能造成电网负荷剧烈波动,危及电网的安全稳定。随着EV入网(V2G)技术的应用,将EV充电站及其周边的分布式新能源发电聚合为虚拟电厂(VPP)后进行优化调度,有助于改善EV用户充放电的经济性及满意度,同时提高分布式新能源的利用率,平抑电网负荷波动,但EV充电站的整体充放电负荷是大量个体EV用户随机行为的聚合,难以用数学模型精确描述。针对包含EV的VPP,提出一种基于深度强化学习的交互式调度框架,以最大化VPP内EV用户的总效益。VPP控制中心作为智能体决策EV个体的充放电动作,无需掌握个体详细模型,而是通过与区域电网环境的交互,不断学习和更新动作策略,从而克服集中式优化方法的局限性。该优化调度框架采用深度确定性策略梯度(DDPG)算法进行求解。仿真结果表明,与集中式优化方法相比,该优化算法提高了各EV用户的效益,并使EV充放电负荷与分布式新能源发电协调配合实现削峰填谷,改善了VPP的整体运行性能。

加固后钢筋混凝土十字形柱斜向抗震性能研究

钢筋混凝土十字形柱的斜向抗震性能是影响结构安全性能的重要因素。文章设计了3种新型约束补强方式加固的钢筋混凝土十字形柱试件,在验证有限元建模方法有效性的基础上模拟了斜向低周往复加载,通过分析试件的滞回曲线、骨架曲线、承载力和延性等抗震性能指标研究了试件的斜向抗震性能,并参考相关规范得出了新型约束补强方式十字形柱的压弯承载力计算公式。结果表明:钢筋网格局部加强使试件的斜向峰值位移提高了7.4%~19.4%;柱端嵌贴碳纤维布(Carbon Fiber Reinforced Polymer,CFRP)板条会使试件峰值承载力、延性系数分别提高了约29.2%~34.2%和7.5%~22.9%;柱端嵌贴CFRP板使试件斜向峰值承载力、延性系数分别提高了约99.9%~105.5%和17.4%~24.8%。3种加固方式均可改善钢筋混凝土十字形柱的斜向抗震性能,改善程度由大到小依次为柱端嵌贴CFRP板、柱端嵌贴CFRP板条和钢筋网格局部加强。

复合材料格栅结构筋条纤维形态与拉伸性能

复合材料格栅结构已大量应用于航空航天飞行器中,探究其力学性能具有重要的工程研究价值。为提高复合材料格栅结构的承载效率,提出一种基于“断筋”处理的纤维形态改善方法,通过有限元分析与试验验证的手段,研究了不同断筋比例对复合材料格栅结构节点处纤维形态和拉伸性能的影响。利用建立的有限元模型分析了断筋处理的复合材料格栅结构拉伸失效机理,仿真与试验结果误差均小于10%。结果表明:在成型过程中适当进行断筋处理,能够显著降低节点处纤维弯曲角度,提高拉伸性能。相较于不进行断筋处理,断筋比例为30%时,拉伸极限载荷提高了24.4%。

基于深度强化学习的微电网在线优化

针对微电网的随机优化调度问题,提出了一种基于深度强化学习的微电网在线优化算法。利用深度神经网络近似状态-动作值函数,把蓄电池的动作离散化作为神经网络输出,然后利用非线性规划求解剩余决策变量并计算立即回报,通过Q学习算法,获取最优策略。为使得神经网络适应风光负荷的随机性,根据风电、光伏和负荷功率预测曲线及其预测误差,利用蒙特卡洛抽样生成多组训练曲线来训练神经网络;训练完成后,保存权重,根据微电网实时输入状态,神经网络能实时输出蓄电池的动作,实现微电网的在线优化调度。在风电、光伏和负荷功率发生波动的情况下与日前优化结果进行对比,验证了该算法相比于日前优化在微电网在线优化中的有效性和优越性。

SGPot:一种基于强化学习的智能电网蜜罐框架

随着工业4.0的快速推进,与之互联的电力数据采集与监视控制(Supervisory Control and Data Acquisition,SCADA)系统逐渐趋于信息化和智能化。由于这些系统本身具有脆弱性以及受到攻击和防御能力的不对等性,使得系统存在各种安全隐患。近年来,针对电力攻击事件频发,亟需提出针对智能电网的攻击缓解方法。蜜罐作为一种高效的欺骗防御方法,能够有效地收集智能电网中的攻击行为。针对现有的智能电网蜜罐中存在的交互深度不足、物理工业过程仿真缺失、扩展性差的问题,设计并实现了一种基于强化学习的智能电网蜜罐框架——SGPot,它能够基于电力行业真实设备中的系统不变量模拟智能变电站控制端,通过电力业务流程的仿真来提升蜜罐欺骗性,诱使攻击者与蜜罐深度交互。为了评估蜜罐框架的性能,搭建了小型智能变电站实验验证环境,同时将SGPot和现有的GridPot以及SHaPe蜜罐同时部署在公网环境中,收集了30天的交互数据。实验结果表明,SGPot收集到的请求数据比GridPot多20%,比SHaPe多75%。SGPot能够诱骗攻击者与蜜罐进行更深度的交互,获取到的交互会话长度大于6的会话数量多于GridPot和SHaPe。

Reinforcement Learning-Empowered Graph Convolutional Network Framework for Data Integrity Attack Detection in Cyber-Physical Systems

The massive integration of communication and information technology with the large-scale power grid has enhanced the efficiency, safety, and economical operation of cyber-physical systems. However, the open and diversified communication environment of the smart grid is exposed to cyber-attacks. Data integrity attacks that can bypass conventional security techniques have been considered critical threats to the operation of the grid. Current detection techniques cannot learn the dynamic and heterogeneous characteristics of the smart grid and are unable to deal with non-euclidean data types. To address the issue, we propose a novel Deep-Q-Network scheme empowered with a graph convolutional network (GCN) framework to detect data integrity attacks in cyber-physical systems. The simulation results show that the proposed framework is scalable and achieves higher detection accuracy, unlike other benchmark techniques.

Automatic Voltage Control of Differential Power Grids Based on Transfer Learning and Deep Reinforcement Learning

In terms of model-free voltage control methods,when the device or topology of the system changes,the model’s accuracy often decreases,so an adaptive model is needed to coordinate the changes of input.To overcome the defects of a model-free control method,this paper proposes an automatic voltage control(AVC)method for differential power grids based on transfer learning and deep reinforcement learning.First,when constructing the Markov game of AVC,both the magnitude and number of voltage deviations are taken into account in the reward.Then,an AVC method based on constrained multiagent deep reinforcement learning(DRL)is developed.To further improve learning efficiency,domain knowledge is used to reduce action space.Next,distribution adaptation transfer learning is introduced for the AVC transfer circumstance of systems with the same structure but distinct topological relations/parameters,which can perform well without any further training even if the structure changes.Moreover,for the AVC transfer circumstance of various power grids,parameter-based transfer learning is created,which enhances the target system’s training speed and effect.Finally,the method’s efficacy is tested using two IEEE systems and two real-world power grids.

钢板铰与钢板带和套管集成加固网架结构方法研究

针对网架结构传统加固方法中存在的施工风险大、工期长及经济指标高的缺陷,提出了一种钢板铰与钢板带和变长套管集成加固方法,该集成加固方法有效解决了网架结构拉杆、节点承载力不足及压杆稳定性不足的问题,形成了一套网架结构加固的新型成套技术,并采用该集成加固方法完成某展览馆网架结构加固工程。加固工程实践表明:该集成加固方法避免了传统网架结构加固中的焊接作业,消除了焊接应力的影响以及焊接过程中钢管高温软化可能导致的非预期破坏,缩短了施工作业时间,取得了良好的社会与经济效益,为以后类似改造工程提供了借鉴。

人机混合增强决策智能在新型电力系统调控中的应用与展望

新型电力系统的调控运行面临规模及随机性升高、海量多元资源协同困难等难题,现有传统优化和人工经验的调控手段难以应对。以强化学习为代表的决策智能技术在表征能力和决策速度方面具备显著优势,但在应用中存在诸多关键瓶颈。而人机混合增强智能(hybrid human-machine intelligence,HHMI)技术具有突破这些瓶颈、支撑高效智能调控的巨大潜力。目前HHMI理论研究尚处于初期,应用不足。为挖掘HHMI的潜力并探索其应用方案,基于新型电力系统调控与应急调整的实际需求,分析决策智能的优势与局限,在此基础上,论述HHMI的总体框架、关键技术、应用方案以及在实际应用中面临的关键问题,并对其未来的实践进行展望,可为HHMI在新型电力系统调控中的研究与应用提供思路。

木丝水泥填充墙-RC框架抗震性能试验研究

为研究木丝水泥填充墙-钢筋混凝土(RC)框架的抗震性能和破坏机理,共制作五个试件进行低周反复试验,其中一个试件为普通砖填充墙-RC框架对比试件,两个试件为开窗洞的木丝水泥填充墙-RC框架,另外两个试件为添加玻璃纤维网格布的木丝水泥填充墙-RC框架,并分析试件的滞回曲线、骨架曲线、延性系数、耗能情况与刚度退化等指标。试验结果表明,木丝水泥填充墙-RC框架的抗震性能良好,刚度退化速度远小于普通砖填充墙-RC框架,耗能能力在试件开裂后超过普通砖填充墙-RC框架;在木丝水泥填充墙-RC框架上开窗洞后会削弱其承载力,但破坏时裂缝只是沿四个窗角分别向四个填充墙角扩展,最终形成“X”形裂缝,而木丝水泥填充墙-RC框架与混凝土之间的粘结面未被破坏,在木丝水泥填充墙-RC框架表面加入玻璃纤维网格布后可以减少墙面裂纹的产生,且提高了极限承载力和延性系数,并抑制了刚度的快速下降。

网架开洞增设支撑改造加固设计典型案例分析

根据改造工艺要求,新建带式输送机栈桥斜向穿过现有煤棚网架,为满足工艺所需,确保安全对既有网架进行加固。总结了网架开洞加固设计的方法,以及现场开洞施工中注意的问题,希望对同类工程提供参考价值。

预应力FRP网格加固空心板梁桥的病害修复效果分析

针对空心板梁桥存在的典型病害问题,采用预应力纤维增强复合材料(FRP)网格进行加固,通过数值模拟分析其对底板开裂及铰缝破损病害的修复效果,讨论了FRP网格种类、用量、布置方式等参数对桥梁承载性能的影响.结果表明:在底板开裂部位横向或纵向布置预应力FRP网格,均可有效控制底板开裂和梁体下挠;在跨中横向布置预应力FRP网格,可有效降低铰缝破损的不利影响;若两类病害均显著存在,可采用横向布置预应力FRP网格进行综合加固;在预应力水平与加固量相同的条件下,玄武岩纤维增强复合材料(BFRP)网格与碳纤维增强复合材料(CFRP)网格的加固效果基本相当.合理布置预应力FRP网格可有效改善空心板梁桥底板开裂和铰缝破损病害,有助于提升其长期受力性能.