Decentralized voltage control of distribution network based on multi-agent reinforcement learning

doi:10.3969/j.issn.2097-0706.2024.10.005

Abstract

Abstract:

The integration of large-scale decentralized resources into the distribution network has changed the traditional power flow distribution， resulting in frequent voltage violations. Model-based voltage control methods require a detailed knowledge of power system network topology and have long computation time， making them unsuitable for real-time voltage control.To address this， this paper proposes a multi-agent online learning strategy for decentralized voltage control in distribution networks， considering asynchronous training. The method considered each photovoltaic （PV） inverter as an agent. First， the agents were partitioned and adjusted， then the voltage reactive power control problem of distribution network was modelled as a Markov decision process. Based on distributed system constraints， a multi-agent reinforcement learning decentralized control framework was used， and agents were trained with a multi-agent deep deterministic policy gradient（MADDPG） algorithm. Once trained， the agents could make decentralized decisions using local information without real-time communication， enabling real-time voltage control and reducing network losses by determining the output plan for the PV inverters. Finally， the effectiveness and robustness of the method were verified through simulation.

Key words: distribution network, multi-agent, voltage decentralized control, multi-agent deep deterministic policy gradient, Markov decision process

CLC Number:

TM714

MA Gang, MA Jian, YAN Yunsong, CHEN Yonghua, LAI Yening, LI Zhukun, TANG Jing. Decentralized voltage control of distribution network based on multi-agent reinforcement learning[J]. Integrated Intelligent Energy, 2024, 46(10): 32-39.

Figures/Tables 11

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Table 1

Fig.7

Fig.8

Fig.9

Fig.10

References 20

[1]	黄维维, 张安安, 敬佳佳, 等. 微电网电压/无功协调控制综述[J]. 电网与清洁能源, 2016, 32(3):50-57.
	HUANG Weiwei, ZHANG An'an, JING Jiajia, et al. An overview on micro‑grid voltage/reactive power coordinated control[J]. Power System and Clean Energy, 2016, 32(3):50-57.
[2]	王林, 孔小民, 周忠玉, 等. 云储能模式下的配电网分布式光伏-储能无功优化方法[J]. 综合智慧能源, 2024, 46(6):44-53. doi: 10.3969/j.issn.2097-0706.2024.06.006
	WANG Lin, KONG Xiaomin, ZHOU Zhongyu, et al, Distributed photovoltaic-energy storage reactive power optimization method for distribution networks under cloud energy storage mode[J]. Integrated Intelligent Energy, 2024, 46(6):44-53. doi: 10.3969/j.issn.2097-0706.2024.06.006
[3]	林泓宏, 余涛, 张桂源, 等. 基于数据驱动的高比例新能源配电网无功优化算法[J]. 综合智慧能源, 2023, 45(11):10-19. doi: 10.3969/j.issn.2097-0706.2023.11.002
	LIN Honghong, YU Tao, ZHANG Guiyuan, et al. Data‑driven reactive power optimization algorithm for the distribution network with high proportion of renewable energy[J]. Integrated Intelligent Energy, 2023, 45(11):10-19. doi: 10.3969/j.issn.2097-0706.2023.11.002
[4]	李鹏, 姜磊, 王加浩, 等. 基于深度强化学习的新能源配电网双时间尺度无功电压优化[J]. 中国电机工程学报, 2023, 43(16):6255-6266.
	LI Peng, JIANG Lei, WANG Jiahao, et al. Optimization of dual‑time scale reactive voltage for distribution network with renewable energy based on deep reinforcement learning[J]. Proceedings of the CSEE, 2023, 43(16):6255-6266.
[5]	郑许林, 袁源, 柏晶晶, 等. 分布式光伏无功电压主动支撑控制方法[J]. 可再生能源, 2023, 41(12):1685-1690.
	ZHENG Xulin, YUAN Yuan, BAI Jingjing, et al. Active support control method of distributed photovoltaic reactive power and voltage[J]. Renewable Energy Resources, 2023, 41(12):1685-1690.
[6]	潘思蓉, 刘友波, 唐志远, 等. 含深度学习代理模型的有源配电网电压无功控制进化算法[J]. 电力系统保护与控制, 2022, 50(17):97-106.
	PAN Sirong, LIU Youbo, TANG Zhiyuan, et al. An evolutionary algorithm for Volt/Var control in an active distribution network with a deep learning surrogate mode[J]. Power System Protection and Control, 2022, 50(17):97-106.
[7]	DUAN J J, SHI D, DIAO R S, et al. deep‑reinforcement‑ learning‑based autonomous voltage control for power grid operations[J]. IEEE Transactions on Power Systems, 2020, 35(1):814-817.
[8]	邵美阳, 吴俊勇, 石琛, 等. 基于数据驱动和深度置信网络的配电网无功优化[J]. 电网技术, 2019, 43(6):1874-1885.
	SHAO Meiyang, WU Junyong, SHI Chen, et al. Reactive power optimization of distribution network based on data driven and deep belief network[J]. Power System Technology, 2019, 43(6):1874-1885.
[9]	ZHANG Y, WAN X N, WANG J H, et al. Deep reinforcement learning based Volt-VAR optimization in smart distribution systems[J]. IEEE Transactions on Smart Grid, 2021. 12(1): 361-371.
[10]	史景坚, 周文涛, 张宁, 等. 含储能系统的配电网电压调节深度强化学习算法[J]. 电力建设, 2020, 41(3):71-78. doi: 10.3969/j.issn.1000－7229.2020.03.009
	SHI Jingjian, ZHOU Wentao, ZHANG Ning, et al. Deep reinforcement learning algorithm of voltage regulation in distribution network with energy storage system[J]. Electric Power Construction, 2020, 41(3):71-78. doi: 10.3969/j.issn.1000－7229.2020.03.009
[11]	FERREIRA L R, AOKI A R, LAMBERT-TORRES G. A reinforcement learning approach to solve service restoration and load management simultaneously for distribution networks[J]. IEEE Access, 2019,7:145978-145987.
[12]	张志荣, 邱晓燕, 孙旭, 等. 基于ADMM的交直流混合配电网分布式电压优化模型[J]. 电网技术, 2021, 45(11):4551-4559.
	ZHANG Zhirong, QIU Xiaoyan, SUN Xu, et al. Distributed voltage optimization model for AC/DC Hybrid distribution network based on ADMM[J]. Power System Technology, 2021, 45(11):4551-4559.
[13]	邓清唐, 胡丹尔, 蔡田田, 等. 基于多智能体深度强化学习的配电网无功优化策略[J]. 电工电能新技术, 2022, 41(2):10-20. doi: 10.12067/ATEEE2103019
	DENG Qingtang, HU Dan'er, CAI Tiantian, et al. Reactive power optimization strategy of distribution network based on multi agent deep reinforcement learning[J]. Advanced Technology of Electrical Engineering and Energy, 2022, 41(2):10-20. doi: 10.12067/ATEEE2103019
[14]	张沛, 朱驻军, 谢桦. 基于深度强化学习近端策略优化的电网无功优化方法[J]. 电网技术, 2023, 47(2):562-572.
	ZHANG Pei, ZHU Zhujun, XIE Hua. Reactive power optimization based on proximal policy optimization of deep reinforcement learning[J]. Power System Technology, 2023, 47(2):562-572.
[15]	LIU H T, WU W C. Online multi‑agent reinforcement learning for decentralized inverter‑based Volt-VAR control[J]. IEEE Transactions on Smart Grid, 2021, 12(4):2980-2990.
[16]	刘硕, 冯斌, 郭创新, 等. 考虑自注意力和时序记忆的深度强化学习配电网无功电压控制策略[J/OL]. 中国电机工程学报,(2023-10-09)[2024-05-06]. http://kns.cnki.net/kcms/detail/11.2107.tm.20231007.1822.004.html.
	LIU Shuo, FENG Bin, GUO Chuangxin, et al. Volt-Var control strategy of distribution network based on deep reinforcement learning considering self‑attention and temporal‑memory[J/OL]. Proceedings of the CSEE,(2023-10-09)[2024-05-06]. http://kns.cnki.net/kcms/detail/11.2107.tm.20231007.1822.004.html.
[17]	KABIRI R, HOLMES D G, MCGRATH B P, et al. LV grid voltage regulation using transformer electronic tap changing, with PV inverter reactive power injection[J]. IEEE Journal of Emerging and Selected Topics in Power Electronics, 2015, 3(4):1182-1192.
[18]	王康平, 张兴科, 刘财华, 等. 基于自适应下垂控制的风电场无功电压控制策略[J]. 综合智慧能源, 2022, 44(4):12-19. doi: 10.3969/j.issn.2097-0706.2022.04.002
	WANG Kangping, ZHANG Xingke, LIU Caihua, et al. Reactive power and voltage control strategy based on adaptive droop control for wind power plants[J]. Integrated Intelligent Energy, 2022, 44(4):12-19. doi: 10.3969/j.issn.2097-0706.2022.04.002
[19]	郑晓莹, 陈政轩, 曾琮. 含分布式电源的配电网双层分区调压策略[J]. 电力系统保护与控制, 2021, 49(6):90-97.
	ZHENG Xiaoying, CHEN Zhengxuan, ZENG Cong. Double‑layer partition voltage regulation strategy of a distribution network with distributed generation[J]. Power System Protection and Control, 2021, 49(6):90-97.
[20]	刘朝阳, 穆朝絮, 孙长银. 深度强化学习算法与应用研究现状综述[J]. 智能科学与技术学报, 2020, 2(4):314-326. doi: 10.11959/j.issn.2096-6652.202034
	LIU Zhaoyang, MU Chaoxu, SUN Changyin. An overview on algorithms and applications of deep reinforcement learning[J]. Chinese Journal of Intelligent Science and Technology, 2020, 2(4):314-326.

控制方法	平均电压偏差（标幺值）	电压可控率/%	总网损/MW	平均时间/ms
无控制	0.639 4	72.35	0.128 6
MADDPG	0.302 2	89.67	0.093 7	2.47
IPPO	0.384 5	80.74	0.149 1	7.36
OLDC-MADDPG	0.285 2	90.43	0.067 8	2.83