基于Transformer嵌入式深度确定性策略梯度法的MB-PSS稳定性控制

doi:10.3969/j.issn.2097-0706.2024.12.001

摘要/Abstract

摘要：

新能源发电出力具有波动性、不确定性特征，会破坏电力系统电压和频率的动态平衡，引发潮流分布的改变，进而造成电网解列，威胁电网的整体稳定与供电安全。多频段电力系统稳定器（MB-PSS）的稳定控制能够提高电力系统电能传输时的可靠性和稳定性。为保证MB-PSS的稳定性，结合深度强化学习中的深度确定性策略梯度（DDPG）法以及生成式预训练模型（GPT）中的Transformer机制，提出了一种Transformer嵌入式DDPG（TDDPG）的多频段电力系统稳定器稳定性控制方法，将Transformer机制融入DDPG法的2个神经网络中，通过编码器和译码器来提高输入参数的维度，以解决DDPG法中2组神经网络输入过少的问题。三相接地故障和两相接地故障仿真结果表明，基于TDDPG法的多频段电力系统稳定器稳定性控制方法训练效果好，具有更高的控制精度。

关键词: 新能源, 多频段电力系统稳定器, 深度强化学习, 生成式预训练模型, 深度确定性策略梯度法, Transformer, 神经网络

Abstract:

The renewable energy generation has volatility and uncertainty characteristics， which can disrupt the dynamic balance of voltage and frequency in power systems， alter power flow distribution， and consequently cause grid splitting， threatening overall grid stability and power supply security. The stability control of the multiband power system stabilizer （MB-PSS） can enhance the reliability and stability of power transmission within power systems. To ensure MB-PSS stability， a Transformer-embedded deep deterministic policy gradient（TDDPG） method was proposed by combining the deep deterministic policy gradient（DDPG） method in deep reinforcement learning with the Transformer mechanism from generative pre-trained models（GPT）. The Transformer mechanism was integrated into two neural networks of the DDPG method， utilizing encoders and decoders to increase the dimensionality of input parameters， addressing the issue of insufficient input in the two neural networks of the DDPG method. Simulation results of three-phase and two-phase grounding faults demonstrate that the multi-band power system stabilizer stability control method based on the TDDPG method achieves excellent training results with higher control precision.

Key words: renewable energy, multiband power system stabilizer, deep reinforcement learning, generative pre-training model, deep deterministic policy gradient method, Transformer, neural network

中图分类号:

TK712：TM712

殷林飞, 赵镱然. 基于Transformer嵌入式深度确定性策略梯度法的MB-PSS稳定性控制[J]. 综合智慧能源, 2024, 46(12): 1-9.

YIN Linfei, ZHAO Yiran. Stability control of multiband power system stabilizer based on Transformer-embedded deep deterministic policy gradient method[J]. Integrated Intelligent Energy, 2024, 46(12): 1-9.

图/表 14

图1

图2

图3

图4

图5

图6

表1

图7

图8

图9

表2

表3

图10

图11

参考文献 34

[1]	伏云辉. 全球150余国提出碳中和目标[J]. 生态经济, 2024, 40(3): 1-4.
[2]	滕佳伦, 李宏仲. 碳中和背景下综合智慧能源的发展现状及关键技术分析[J]. 综合智慧能源, 2023, 45(8): 53-63. doi: 10.3969/j.issn.2097-0706.2023.08.007
	TENG Jialun, LI Hongzhong. Analysis on development and key technologies of integrated intelligent energy in the context of carbon neutrality[J]. Integrated Intelligent Energy, 2023, 45(8): 53-63. doi: 10.3969/j.issn.2097-0706.2023.08.007
[3]	于永. 新能源电力系统中的储能技术探讨[J]. 智慧中国, 2022(11): 80-81.
[4]	江崇熙, 石鹏, 黄伟, 等. 考虑多振荡模式的多频段电力系统稳定器参数整定方法[J]. 电力系统自动化, 2020, 44(4): 142-149.
	JIANG Chongxi, SHI Peng, HUANG Wei, et al. Parameter setting method for multi-band power system stabilizer considering multiple oscillation modes[J]. Automation of Electric Power Systems, 2020, 44(4): 142-149.
[5]	李朋真, 贾冰珂, 刘艳红, 等. 燃烧后二氧化碳捕集系统的改进自抗扰控制[J]. 综合智慧能源, 2023, 45(8): 18-25. doi: 10.3969/j.issn.2097-0706.2023.08.003
	LI Pengzhen, JIA Bingke, LIU Yanhong, et al. Modified active disturbance rejection control on the post-combustion CO₂ capture system[J]. Integrated Intelligent Energy, 2023, 45(8): 18-25. doi: 10.3969/j.issn.2097-0706.2023.08.003
[6]	CHANG Jinwei, LI Zhi, HUANG Yan, et al. Multi-objective optimization of a novel combined cooling, dehumidification and power system using improved M-PSO algorithm[J]. Energy, 2022,239: 122487.
[7]	李明扬, 窦梦园. 基于强化学习的含电动汽车虚拟电厂优化调度[J]. 综合智慧能源, 2024, 46(6): 27-34. doi: 10.3969/j.issn.2097-0706.2024.06.004
	LI Mingyang, DOU Mengyuan. Optimal scheduling of virtual power plants integrating electric vehicles based on reinforcement learning[J]. Integrated Intelligent Energy, 2024, 46(6): 27-34. doi: 10.3969/j.issn.2097-0706.2024.06.004
[8]	胡泽, 朱子晴, 卜思齐, 等. 基于深度强化学习的区域综合能源定价策略研究[J]. 综合智慧能源, 2023, 45(7): 87-96. doi: 10.3969/j.issn.2097-0706.2023.07.010
	HU Ze, ZHU Ziqing, BU Siqi, et al. Pricing strategy in district-level integrated energy market based on deep reinforcement learning[J]. Integrated Intelligent Energy, 2023, 45(7): 87-96. doi: 10.3969/j.issn.2097-0706.2023.07.010
[9]	袁雪峰, 马进, 强硕, 等. 遗传算法优化锅炉汽包水位不完全微分PID参数[J]. 华电技术, 2018, 40(10): 7-11.
	YUAN Xuefeng, MA Jin, QIANG Shuo, et al. Incomplete differential PID parameters of boiler drum water level optimized by genetic algorithm[J]. Huadian Technology, 2018, 40(10): 7-11.
[10]	康岩松, 臧顺来. 基于多种策略的改进粒子群优化算法[J]. 东北大学学报(自然科学版), 2023, 44(8): 1089-1097.
	KANG Yansong, ZANG Shunlai. Improved particle swarm optimization algorithm based on multiple strategies[J]. Journal of Northeastern University (Natural Science), 2023, 44(8): 1089-1097.
[11]	刘子祺, 苏婷婷, 何佳阳, 等. 基于多目标粒子群算法的配电网储能优化配置研究[J]. 综合智慧能源, 2023, 45(6): 9-16. doi: 10.3969/j.issn.2097-0706.2023.06.002
	LIU Ziqi, SU Tingting, HE Jiayang, et al. Research on the optimal allocation of energy storage in distribution network based on multi-objective particle swarm optimization algorithm[J]. Integrated Intelligent Energy, 2023, 45(6): 9-16. doi: 10.3969/j.issn.2097-0706.2023.06.002
[12]	冯伟杨, 林思雨, 冯婧涛, 等. 基于Q学习的蜂窝车联网边缘计算系统PC-5/Uu接口联合卸载策略[J]. 电子学报, 2024, 52(2): 385-395. doi: 10.12263/DZXB.20220922
	FENG Weiyang, LIN Siyu, FENG Jingtao, et al. Q-learning based joint PC-5/Uu offloading strategy for C-V2X based vehicular edge computing system[J]. Acta Electronica Sinica, 2024, 52(2): 385-395. doi: 10.12263/DZXB.20220922
[13]	郑庆明, 井延伟, 梁涛, 等. 基于DDPG算法的离网型可再生能源大规模制氢系统优化调度[J]. 综合智慧能源, 2024, 46(6): 35-43. doi: 10.3969/j.issn.2097-0706.2024.06.005
	ZHENG Qingming, JING Yanwei, LIANG Tao, et al. Optimized scheduling on large-scale hydrogen production system for off-grid renewable energy based on DDPG algorithm[J]. Integrated Intelligent Energy, 2024, 46(6): 35-43. doi: 10.3969/j.issn.2097-0706.2024.06.005
[14]	范培潇, 柯松, 杨军, 等. 基于改进多智能体深度确定性策略梯度的多微网负荷频率协同控制策略[J]. 电网技术, 2022, 46(9): 3504-3515.
	FAN Peixiao, KE Song, YANG Jun, et al. Load frequency coordinated control strategy of multi-microgrid based on improved MA-DDPG[J]. Power System Technology, 2022, 46(9): 3504-3515.
[15]	展明星, 王致诚, 李致远. 基于模糊自适应PID和BP神经网络PID控制的永磁同步电机控制效果比较研究[J]. 广西电力, 2022, 45(1): 26-32.
	ZHAN Mingxing, WANG Zhicheng, LI Zhiyuan. Comparative study on the control effect of permanent magnet synchronous motors based on fuzzy adaptive PID and BP neural network PID control[J]. Guangxi Electric Power, 2022, 45(1): 26-32.
[16]	焦建利. ChatGPT助推学校教育数字化转型: 人工智能时代学什么与怎么教[J]. 中国远程教育, 2023(4): 16-23.
	JIAO Jianli. ChatGPT boosting digital transformation of education in schooling: What to learn and how to teach in the era of artificial intelligence[J]. Chinese Journal of Distance Education, 2023(4): 16-23.
[17]	TIAN D, LI M C, REN Q B, ZHANG X J, et al. Intelligent question answering method for construction safety hazard knowledge based on deep semantic mining[J]. Automation in Construction, 2023,145: 104670.
[18]	YAN Z M, XU Y. Real-time optimal power flow with linguistic stipulations: Integrating GPT-agent and deep reinforcement learning[J]. IEEE Transactions on Power Systems, 2024, 39(2): 4747-4759.
[19]	HAMED R N, ABOLFAZL Z, HOLGER V. Actor-critic learning based PID control for robotic manipulators[J]. Applied Soft Computing, 2024,151: 111153.
[20]	JIANG H Y, LU Y Y, ZHANG D N, et al. Deep learning-based fusion networks with high-order attention mechanism for 3D object detection in autonomous driving scenarios[J]. Applied Soft Computing, 2024,152:111253.
[21]	BARHAGHTALAB M H, SEPESTANAKI M A, MOBAYEN S, et al. Design of an adaptive fuzzy-neural inference system-based control approach for robotic manipulators[J]. Applied Soft Computing, 2023,149: 110970.
[22]	ZHANG G Z, HU W H, ZHAO J B, et al. A novel deep reinforcement learning enabled multi-band PSS for multi-mode oscillation control[J]. IEEE Transactions on Power Systems, 2021, 36(4): 3794-3797.
[23]	王丽萍, 任宇, 邱启仓, 等. 多目标进化算法性能评价指标研究综述[J]. 计算机学报, 2021, 44(8): 1590-1619.
	WANG Liping, REN Yu, QIU Qicang, et al. Survey on performance indicators for multi-objective evolutionary algorithms[J]. Chinese Journal of Computers, 2021, 44(8): 1590-1619.
[24]	ZHANG X S, GUO Z X, PAN F, et al. Dynamic carbon emission factor based interactive control of distribution network by a generalized regression neural network assisted optimization[J]. Energy, 2023, 283: 129132.
[25]	ASHRAFUL H, SAIFUR R. Short-term electrical load forecasting through heuristic configuration of regularized deep neural network[J]. Applied Soft Computing Journal, 2022, 122: 108877.
[26]	HAN S, ZHOU W B, LU S, et al. Regularly updated deterministic policy gradient algorithm[J]. Knowledge-Based Systems, 2021,214: 106736.
[27]	张俊杰, 张聪, 赵涵捷. 重复利用状态值的竞争深度Q网络算法[J]. 计算机工程与应用, 2021, 57(4): 134-140. doi: 10.3778/j.issn.1002-8331.2007-0125
	ZHANG Junjie, ZHANG Cong, ZHAO Hanjie. Dueling deep Q network algorithm with state value reuse[J]. Computer Engineering and Applications, 2021, 57(4): 134-140. doi: 10.3778/j.issn.1002-8331.2007-0125
[28]	王壮. 非高斯噪声下基于有界非线性函数的快速盲均衡方法[J]. 通信技术, 2020, 53(7): 1606-1611.
	WANG Zhuang. Fast blind equalization based on bounded non-linear function under non-Gaussian noise[J]. Communications Technology, 2020, 53(7): 1606-1611.
[29]	GU Z Y, SHE C Y, HARDJAWANA W, et al. Knowledge-assisted deep reinforcement learning in 5G scheduler design: from theoretical framework to implementation[J]. IEEE Journal on Selected Areas in Communications, 2021, 39(7): 2014-2028.
[30]	KOBAYASHI T, ILBOUDO W E L. T-soft update of target network for deep reinforcement learning[J]. Neural Networks, 2021,136: 63-71.
[31]	ZHANG X S, LI C Z, XU B, et al. Dropout deep neural network assisted transfer learning for bi-objective Pareto AGC dispatch[J]. IEEE Transactions on Power Systems, 2022, 38(2): 1432-1444.
[32]	WEI Q, MA H L, CHEN C L, et al. Deep reinforcement learning with quantum-inspired experience replay[J]. IEEE Transactions on Cybernetics, 2022, 52(9): 9326-9338.
[33]	ZHAO L P, ALHOSHAN W, FERRARI A, et al. Natural language processing for requirements engineering: A systematic mapping study[J]. ACM Computing Surveys, 2022, 54(3): 1-41.
[34]	朱张莉, 饶元, 吴渊, 等. 注意力机制在深度学习中的研究进展[J]. 中文信息学报, 2019, 33(6): 1-11.
	ZHU Zhangli, RAO Yuan, WU Yuan, et al. Research progress of attention mechanism in deep learning[J]. Journal of Chinese Information Processing, 2019, 33(6):1-11.

算法	参数
PID	K_P=50，K_I=-3.834 20，K_D=0.104 32，K_N=100
PPO	α=0.008，β=0.5，γ=0.95，Action={30,200,-10,10,0,10}
SAC	α=0.001，β=0.5，γ=0.99，Action={0,100,-10,1,-1,1}
TD3	α=0.001，β=0.5，γ=0.99，Action={-100,100,-5,5,-10,10}
DDPG	α=0.001，β=0.3，γ=1.00，Action={40,200,-1,1,0,10}
TDDPG	α=0.001，β=0.5，γ=0.99，Action={30,200,-10,10,-10,10}
LMS	ω=1，φ=0.01

算法	IAE/V	ITAE /V	ISE/V²	ITSE/V²
TDDPG	0.649 8	5.895	0.041 36	0.246 7
DDPG	0.879 7	7.193	0.045 79	0.469 1
PID	0.802 9	6.336	0.046 30	0.290 5
SAC	1.153 0	11.650	0.080 13	0.715 9
TD3	∞	∞	∞	∞
PPO	∞	∞	∞	∞
LMS	0.954 1	9.023	0.056 59	0.493 3

算法	IAE /V	ITAE /V	ISE/V²	ITSE/V²
TDDPG	0.614 7	6.334	0.043 40	0.366 7
DDPG	0.839 7	7.593	0.045 60	0.405 3
PID	0.850 8	7.991	0.044 30	0.443 3
SAC	1.122 0	10.900	0.071 34	0.617 2
TD3	∞	∞	∞	∞
PPO	∞	∞	∞	∞
LMS	0.903 1	8.433	0.051 59	0.378 0