基于MADDPG算法的建筑群柔性负荷优化调控方法

doi:10.3969/j.issn.2097-0706.2023.07.007

综合智慧能源 ›› 2023, Vol. 45 ›› Issue (7): 61-69.doi: 10.3969/j.issn.2097-0706.2023.07.007

基于MADDPG算法的建筑群柔性负荷优化调控方法

包义辛¹(), 徐椤赟¹^,²(), 杨强¹^,^*()

1.浙江大学电气工程学院，杭州 310027
2.浙江省白马湖实验室有限公司，杭州 310056

收稿日期:2023-06-05 修回日期:2023-07-03 接受日期:2023-07-25 出版日期:2023-07-25
通讯作者: *杨强（1979），男，教授，博士，博士生导师，从事从事综合能源系统规划与运行控制研究，qyang@zju.edu.cn。
作者简介:包义辛（2001），男，在读硕士研究生，从事电力系统优化规划与运行调控等方面的研究，3190103975@zju.edu.cn；
徐椤赟（1993），男，博士，从事新能源微电网高效运行与稳定性等方面的研究，luoyun.xu@outlook.com。
基金资助:
中国电机工程学会青年人才托举工程项目(CSEE-YESS-2021020)

Optimized control method for flexible load of a building complex based on MADDPG reinforcement learning

BAO Yixin¹(), XU Luoyun¹^,²(), YANG Qiang¹^,^*()

1. College of Electrical Engineering，Zhejiang University，Hangzhou 310027 China
2. Zhejiang Baima Lake Laboratory Company Limited，Hangzhou 310056， China

Received:2023-06-05 Revised:2023-07-03 Accepted:2023-07-25 Published:2023-07-25
Supported by:
The Young Elite Scientists Sponsorship Program by CSEE(CSEE-YESS-2021020)

摘要/Abstract

摘要：

随着电网调度环境和信息整理环境日趋复杂，电网调控的难度也随之增加。针对深度强化学习技术具有有效感知复杂系统运行状态、适应性强、可扩展性好等特点，提出了基于深度强化学习的配网优化调度方法。构建了考虑源-网-荷-储的模拟建筑体配网模型，从原理出发对多智能体深度确定性策略梯度（MADDPG）算法进行静态优化，将模型与真实数据导入适用于电网级目标的多智能体强化学习框架中，尝试用优化后的算法对配网系统进行电压调控。结果表明，所用算法基本消除了配网系统的违规峰值电压，降低了总体电压偏差；优化后的多目标导向算法在保持电压稳定的同时减小了负载-发电功率差，使负载功率损耗维持较低水平，表明基于深度强化学习的建筑群柔性负荷优化调控方法具有一定有效性。

关键词: 微电网调控, 能量管理, 深度强化学习, 确定性策略梯度, 多目标优化, 源网荷储, 建筑群柔性负荷

Abstract:

The power grid dispatch environment and information organization environment have become more complex， and the difficulty of power grid regulation has gradually increased. Since deep reinforcement learning technology is of effective perception on complex system operation statuses，strong adaptability and good scalability，a distribution network optimization scheduling method based on deep reinforcement learning is proposed. Based on the simulated source-network-load-storage integrated distribution network model of a building complex，Multi-Agent Deep Deterministic Policy Gradient（MADDPG）algorithm was statically optimized from its principle.The model and real data were input into a multi-agent reinforcement learning framework suitable for grid-level objectives，and the optimized algorithm was tried to regulate the voltage of the distribution network system. The results show that the algorithm basically eliminates the abnormal peak voltages and reduces the overall voltage deviation.The optimized multi-objective oriented algorithm reduces the load-generated power difference while levelling the voltage off at a low level. The optimized control method for building complex flexible load based on reinforcement learning is proven to be effective.

Key words: microgrid regulation, energy management, deep reinforcement learning, deterministic policy gradient, multi-objective optimization, source-grid-load- storage, flexible load of buildings

中图分类号:

TK01⁺8：TU18

包义辛, 徐椤赟, 杨强. 基于MADDPG算法的建筑群柔性负荷优化调控方法[J]. 综合智慧能源, 2023, 45(7): 61-69.

BAO Yixin, XU Luoyun, YANG Qiang. Optimized control method for flexible load of a building complex based on MADDPG reinforcement learning[J]. Integrated Intelligent Energy, 2023, 45(7): 61-69.

图/表 12

表1

图1

图2

表2

表3

图3

图4

图5

表4

图6

图7

图8

参考文献 31

[1]	王晓燕, 宋方宇轩, 卢珊. 电网调控中人工智能应用的关键技术研究[J]. 科技与创新, 2022,(23):4-6,11
	WANG Xiaoyan, SONG Fangyuxuan, LU Shan. Research on key technologies of artificial intelligence application in power grid regulation[J]. Science and Technology & Innovation, 2022,(23):4-6,11.
[2]	崔文虎. 电力系统故障演化建模与分析[D]. 成都: 中国电子科技大学, 2018.
	CUI Wenhu. Power system fault evolution modeling and analysis[D]. Chengdu: University of Electronic Science and Technology of China, 2018.
[3]	姜丽珍, 董淑杰, 闫振伟, 等. 电网调控技术在电力系统中的应用[J]. 电子制作, 2019(12):69-70.
	JIANG Lizhen, DONG Shujie, YAN Zhenwei, et al. Application of power grid regulation technology in power system[J]. Practical Electronics, 2019(12):69-70.
[4]	吉斌, 孙绘, 昌力, 等. 黏性电力用户参与需求侧响应的行为决策建模与分析[J]. 综合智慧能源, 2022, 44(2):80-88. doi: 10.3969/j.issn.2097-0706.2022.02.011
	JI Bing, SUN Hui, CHANG Li, et al. Modeling and analysis on decision making behavior of loyal users participating in demand-side response[J]. Intrgrated Intelligent Energy, 2022, 44(2):80-88.
[5]	闪鑫, 陆晓, 翟明玉, 等. 人工智能应用于电网调控的关键技术分析[J]. 电力系统自动化, 2019, 43(1):49-57.
	SHAN Xin, LU Xiao, ZHAI Mingyu, et al. Analysis of key technologies for artificial intelligence applied to power grid dispatch and control[J]. Automation of Electric Power Systems, 2019, 43(1):49-57.
[6]	LIU Y, HUANG X, LI S, et al. A construction method of power grid monitoring knowledge graph[J]. Journal of Physics:Conference Series, 2022, 2166(1):12-14.
[7]	赵俊峰, 庄哲寅, 承轶青, 等. 资源描述框架语义网视角下的智能电网模型[J]. 华电技术, 2014, 36(4):19-21.
	ZHAO Junfeng, ZHUANG Zheyin, CHENG Yiqing, et al. Intelligent power grid model under visual angle of RDF Semantic Web[J]. Huadian Technology, 2014, 36(4): 19-21.
[8]	CHEN Z, MI W, LIN J, et al. Discussion on intelligence assistant scheme of dispatching and control operation in power grid[J]. Automation of Electric Power Systems, 2019, 43(22):173-178.
[9]	吴俊勇. 国内外智能电网的发展战略[J]. 变频器世界, 2011, 34(9):36-37.
	WU Junyong. Development strategy of smart grid at China and abroad[J]. The World of Inverters, 2011, 34(9):36-37.
[10]	巫飞新. 国内外智能电网技术发展现状[J]. 电气开关, 2012, 50(2):3-6.
	WU Feixin. Development status of smart grid technology home and abroad[J]. Electric Switchgear, 2012, 50(2):3-6.
[11]	史梦, 张志生, 罗学礼, 曹敏. 智能电网建设思路及国内外发展战略[C]// 2010年云南电力技术论坛论文集(文摘部分), 2010:1571-1575.
[12]	FANG J, WANG Y, LEI Z, et al. Control Strategy and Performance Analysis of Electrochemical Energy Storage Station Participating in Power System Frequency Regulation:A case study of the jiangsu power grid[J]. Sustainability, 2022, 14(15):9189. doi: 10.3390/su14159189
[13]	HUANG Y, LI H, WANG Z, et al. Research on the mechanism and forecast opower grid regulation policy under the background of new electricity reform in China[J]. Journal of Physics:Conference Series, 2019, 1176(4):2030.
[14]	李明节, 陶洪铸, 许洪强, 等. 电网调控领域人工智能技术框架与应用展[J]. 电网技术, 2020, 44(2):393-400.
	LI Mingjie, TAO Hongzhu, XU Hongqiang, et al. The technical framework and application prospect of artificial intelligence application in the field of power grid dispatching and control[J]. Power System Technology, 2020, 44(2):393-400.
[15]	ELFAKI, ABDELRAHMAN O, SIM L, et al. Designing learning object repository using first order logic[C]// 2nd IIAI International Conference on Advanced Applied Informatics,IIAI-AAI 2013,413-414, 2013.
[16]	范士雄, 李立新, 王松岩, 等. 人工智能技术在电网调控中的应用研究[J]. 电网技术, 2020, 44(2):401-411.
	FAN Shixiong, LI Lixin, WANG Songyan, et al. Bowen.application analysis and exploration of artificial intelligence technology in power grid dispatch and control[J]. Power System Technology, 2020, 44(2):401-411.
[17]	CLAESSENS B, VRANCX P, RUELENS F. Convolutional neural networks for automatic state-time feature extraction in reinforcement learning applied to residential load control[J], IEEE Transactions on Smart Grid, 2016, 9(4):3259-3269. doi: 10.1109/TSG.2016.2629450
[18]	李志. 智能电网调度控制系统的运行安全策略[J]. 集成电路应用, 2022, 39(11):132-133.
	LI Zhi. Strategy of operation security of smart grid dispatching control system[J]. Application of IC, 2022, 39(11):132-133.
[19]	LIU S, GAO Y, YANG H, et al. Optimal guidance strategy for flexible load based on hybrid direct load control and time of use[J]. Global Energy Interconnection, 2023, 6(3):297-307. doi: 10.1016/j.gloei.2023.06.004
[20]	AISLING P, CONSTANCE C, KYRI B, et al. GridLearn:Multiagent reinforcement learning for grid-aware building energy management[J]. Electric Power Systems Research, 2022,213.
[21]	华咏竹, 谢强强, 秦会斌, 等. 计及用户端调节容量的变频空调自适应电压调控策略[J]. 综合智慧能源, 2022, 44(2):21-28. doi: 10.3969/j.issn.2097-0706.2022.02.004
	HUA Yongzhu, XIE Qiangqiang, QING Huibin, et al. Adaptive voltage regulation strategy for inverter air conditioners considering the regulation capacity on user side[J]. Intrgrated Intelligent Energy, 2022, 44(2):21-28.
[22]	童家麟, 洪庆, 吕洪坤, 等. 电源侧储能技术发展现状及应用前景综述[J]. 华电技术, 2021, 43(7):17-23.
	TONG Jialin, HONG Qing, LYU Hongkun, et al. Development status and application prospect of power side energy storage technology[J]. Huadian Technology, 2021, 43(7):17-23.
[23]	赵建立, 汤卓凡, 王桂林, 等. 具有储能作用的用户侧资源运行特性[J]. 综合智慧能源, 2022, 44(2):8-14. doi: 10.3969/j.issn.2097-0706.2022.02.002
	ZHAO Jianli, TANG Zhuofan, WANG Guilin, et al. Operation characteristics of user-side resources with energy storage function[J]. Intrgrated Intelligent Energy, 2022, 44(2):8-14.
[24]	LEON T, ALEXANDER S, FLORIAN S, et al. Pandapower an open-source python tool for convenient modeling, analysis,and optimization of electric power systems[J]. IEEE Transactions on Power Systems, 2018, 33(6):6510-6521. doi: 10.1109/TPWRS.59
[25]	DOLATABADI S, MAEDEH G, PIERLUIGI S, et al. An enhanced IEEE 33 bus benchmark test system for distribution system studies[J]. IEEE Transactions on Power Systems, 2021, 36(3):2565-2572. doi: 10.1109/TPWRS.2020.3038030
[26]	BUŞONIU L, BABUŠKA R, BART D. A comprehensive survey of multiagent reinforcement learning[J]. IEEE Transactions on Systems,Man,and Cybernetics, 2008, 310(2):156-172.
[27]	TAN M. Multi-agent reinforcement learning: Independent vs.cooperative agents[C]// Proceedings of the Tenth International Conference on Machine Learning,ICML 1993,330-337
[28]	WILLIAMS J. Simple statistical gradient-following algorithms for connectionist reinforcement learning[J]. Machine Learning, 1992, 8(3):229-256.
[29]	谢昕怡, 应黎明, 田书圣, 等. 基于MADDPG和智能合约的微电网交易决策优化[J]. 电力建设, 2022, 43(11):142-150. doi: 10.12204/j.issn.1000-7229.2022.11.014
	XIE Xingyi, YING Liming, TIAN Shushen, et al. Optimization of microgrid trading strategy based on maddpg and smart contracts[J]. Electric Power Construction, 2022, 43(11):142-150. doi: 10.12204/j.issn.1000-7229.2022.11.014
[30]	DAVID B, ZHANG X, DYLAN W, et al. Power gridworld: A framework for multi-agentreinforcement learning in power systems[C]// Proceedings of the 2022 13th ACM International Conference on Future Energy Systems,Energy, 2022,565-570.
[31]	孟安波, 王鹏, 丁伟锋, 等. 基于强化学习及纵横交叉粒子群算法的电网最优潮流计算[J]. 华电技术, 2021, 43(8):74-82.
	MENG Anbo, WANG Peng, DING Weifeng, et al. Optimal power flow calculation of power grid based on reinforcement learning and crisscross PSO algorithm particle swarm optimization[J]. Huadian Technology, 2021, 43(8):74-82.

建筑类型	具体建筑用途	当前类型数量
1	中型写字楼	4
2	快餐厅	37
3	零售商店	5
4	购物商店	1
5,6,7,8,9	多户住宅	145

可控单元	参数	值
光伏系统	太阳能板容量/kW	40
热水供应系统	热泵性能系数	0.22
	冷/热储水罐容量（标准单位）	1.5/0
	冷/热储水罐热损失系数	0.006/0.008
	加热/降温目标度/℃	45/9
	电热水器效率	0.92
储能系统	电池容量/（kW·h）	75
	电池效率	0.9
	额定功率/kW	30
	功率效率曲线	[0，0.83][0.3，0.85]
	容量功率曲线	[0.0，1][0.8，0.95]

参数	默认值
经验回放缓冲区容量	750
经验抽取数	32
折扣因子	0.95
目标网络更新权值	0.01
actor网络学习率	0.001
critic网络学习率	0.001
训练前数据收集轮次	3
动态学习率调整系数	0.3

项目	调控前	调控后
0.99~1.01电压比例/%	35.458	46.557
平均值	1.003 14	1.003 03
样本方差	0.000 360 08	0.000 337 17

基于MADDPG算法的建筑群柔性负荷优化调控方法

Optimized control method for flexible load of a building complex based on MADDPG reinforcement learning

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 12

参考文献 31

相关文章 15

编辑推荐

Metrics

本文评价

[1]	王泽宁, 李文中, 李东辉, 徐泰山, 俞俊. 基于软件定义的新型电力系统分层自治电力平衡模式研究[J]. 综合智慧能源, 2024, 46(7): 1-11.
[2]	何方波, 裴力耕, 郑睿, 范康健, 张晓曼, 李更丰. “源网荷储”协同助力陕西省新型电力系统建设[J]. 综合智慧能源, 2024, 46(7): 40-46.
[3]	李明扬, 窦梦园. 基于强化学习的含电动汽车虚拟电厂优化调度[J]. 综合智慧能源, 2024, 46(6): 27-34.
[4]	郑庆明, 井延伟, 梁涛, 柴露露, 吕梁年. 基于DDPG算法的离网型可再生能源大规模制氢系统优化调度[J]. 综合智慧能源, 2024, 46(6): 35-43.
[5]	俞胜, 周霞, 沈希澄, 戴剑丰, 刘增稷. 考虑网络攻击影响的源网荷储系统风险评估[J]. 综合智慧能源, 2024, 46(5): 41-49.
[6]	陈勇, 肖雷鸣, 王井南, 吴健. 基于场景扩充的低碳综合能源系统高可靠性容量规划方法[J]. 综合智慧能源, 2024, 46(4): 24-33.
[7]	汤梓涵, 王帅杰, 鞠振河, 雷志奇. 光伏/光热耦合空气源热泵系统性能优化[J]. 综合智慧能源, 2024, 46(4): 34-41.
[8]	孙健, 张云帆, 蔡潇龙, 刘鼎群. 基于预测负荷的暖通空调系统优化调度[J]. 综合智慧能源, 2024, 46(3): 12-19.
[9]	陆文甜. 基于增量交换的主动配电网分布式多目标最优潮流[J]. 综合智慧能源, 2024, 46(2): 43-48.
[10]	孔慧超, 王文钟, 雷一, 彭静, 李海波. 园区受端新型电力系统电力电量再平衡方法[J]. 综合智慧能源, 2024, 46(2): 68-74.
[11]	鲍海波, 梁浚杰, 李想. 园区供电系统广义负荷需求响应建模与分析[J]. 综合智慧能源, 2024, 46(1): 11-17.
[12]	张力, 金立, 任炬光, 刘小兵. 计及气象因素与分时电价影响的综合能源系统负荷调控策略研究[J]. 综合智慧能源, 2024, 46(1): 18-27.
[13]	王永真, 韩艺博, 韩恺, 韩俊涛, 宋阔, 张兰兰. 基于知识图谱的数据中心综合能源系统研究综述[J]. 综合智慧能源, 2023, 45(7): 1-10.
[14]	胡泽, 朱子晴, 卜思齐, 陈家荣, 魏翔. 基于深度强化学习的区域综合能源定价策略研究[J]. 综合智慧能源, 2023, 45(7): 87-96.
[15]	窦真兰, 沈建忠, 张春雁, 江晶晶, 陈祺, 陈婧. 考虑供需不确定性的区域综合能源系统时间解耦分层能量管理[J]. 综合智慧能源, 2023, 45(6): 17-24.