基于MADDPG算法的建筑群柔性负荷优化调控方法

doi:10.3969/j.issn.2097-0706.2023.07.007

综合智慧能源 ›› 2023, Vol. 45 ›› Issue (7): 61-69.doi: 10.3969/j.issn.2097-0706.2023.07.007

基于MADDPG算法的建筑群柔性负荷优化调控方法

包义辛¹(), 徐椤赟¹^,²(), 杨强¹^,^*()

1.浙江大学电气工程学院，杭州 310027
2.浙江省白马湖实验室有限公司，杭州 310056

收稿日期:2023-06-05 修回日期:2023-07-03 接受日期:2023-07-25 出版日期:2023-07-25 发布日期:2023-07-25
通讯作者: *杨强（1979），男，教授，博士，博士生导师，从事从事综合能源系统规划与运行控制研究，qyang@zju.edu.cn。
作者简介:包义辛（2001），男，在读硕士研究生，从事电力系统优化规划与运行调控等方面的研究，3190103975@zju.edu.cn；
徐椤赟（1993），男，博士，从事新能源微电网高效运行与稳定性等方面的研究，luoyun.xu@outlook.com。
基金资助:
中国电机工程学会青年人才托举工程项目(CSEE-YESS-2021020)

Optimized control method for flexible load of a building complex based on MADDPG reinforcement learning

BAO Yixin¹(), XU Luoyun¹^,²(), YANG Qiang¹^,^*()

1. College of Electrical Engineering，Zhejiang University，Hangzhou 310027 China
2. Zhejiang Baima Lake Laboratory Company Limited，Hangzhou 310056， China

Received:2023-06-05 Revised:2023-07-03 Accepted:2023-07-25 Online:2023-07-25 Published:2023-07-25
Supported by:
The Young Elite Scientists Sponsorship Program by CSEE(CSEE-YESS-2021020)

摘要/Abstract

摘要：

随着电网调度环境和信息整理环境日趋复杂，电网调控的难度也随之增加。针对深度强化学习技术具有有效感知复杂系统运行状态、适应性强、可扩展性好等特点，提出了基于深度强化学习的配网优化调度方法。构建了考虑源-网-荷-储的模拟建筑体配网模型，从原理出发对多智能体深度确定性策略梯度（MADDPG）算法进行静态优化，将模型与真实数据导入适用于电网级目标的多智能体强化学习框架中，尝试用优化后的算法对配网系统进行电压调控。结果表明，所用算法基本消除了配网系统的违规峰值电压，降低了总体电压偏差；优化后的多目标导向算法在保持电压稳定的同时减小了负载-发电功率差，使负载功率损耗维持较低水平，表明基于深度强化学习的建筑群柔性负荷优化调控方法具有一定有效性。

关键词: 微电网调控, 能量管理, 深度强化学习, 确定性策略梯度, 多目标优化, 源网荷储, 建筑群柔性负荷

Abstract:

The power grid dispatch environment and information organization environment have become more complex， and the difficulty of power grid regulation has gradually increased. Since deep reinforcement learning technology is of effective perception on complex system operation statuses，strong adaptability and good scalability，a distribution network optimization scheduling method based on deep reinforcement learning is proposed. Based on the simulated source-network-load-storage integrated distribution network model of a building complex，Multi-Agent Deep Deterministic Policy Gradient（MADDPG）algorithm was statically optimized from its principle.The model and real data were input into a multi-agent reinforcement learning framework suitable for grid-level objectives，and the optimized algorithm was tried to regulate the voltage of the distribution network system. The results show that the algorithm basically eliminates the abnormal peak voltages and reduces the overall voltage deviation.The optimized multi-objective oriented algorithm reduces the load-generated power difference while levelling the voltage off at a low level. The optimized control method for building complex flexible load based on reinforcement learning is proven to be effective.

Key words: microgrid regulation, energy management, deep reinforcement learning, deterministic policy gradient, multi-objective optimization, source-grid-load- storage, flexible load of buildings

中图分类号:

TK01⁺8：TU18

包义辛, 徐椤赟, 杨强. 基于MADDPG算法的建筑群柔性负荷优化调控方法[J]. 综合智慧能源, 2023, 45(7): 61-69.

BAO Yixin, XU Luoyun, YANG Qiang. Optimized control method for flexible load of a building complex based on MADDPG reinforcement learning[J]. Integrated Intelligent Energy, 2023, 45(7): 61-69.

图/表 12

表1

图1

图2

表2

表3

图3

图4

图5

表4

图6

图7

图8

参考文献 31

[1]	王晓燕, 宋方宇轩, 卢珊. 电网调控中人工智能应用的关键技术研究[J]. 科技与创新, 2022,(23):4-6,11
	WANG Xiaoyan, SONG Fangyuxuan, LU Shan. Research on key technologies of artificial intelligence application in power grid regulation[J]. Science and Technology & Innovation, 2022,(23):4-6,11.
[2]	崔文虎. 电力系统故障演化建模与分析[D]. 成都: 中国电子科技大学, 2018.
	CUI Wenhu. Power system fault evolution modeling and analysis[D]. Chengdu: University of Electronic Science and Technology of China, 2018.
[3]	姜丽珍, 董淑杰, 闫振伟, 等. 电网调控技术在电力系统中的应用[J]. 电子制作, 2019(12):69-70.
	JIANG Lizhen, DONG Shujie, YAN Zhenwei, et al. Application of power grid regulation technology in power system[J]. Practical Electronics, 2019(12):69-70.
[4]	吉斌, 孙绘, 昌力, 等. 黏性电力用户参与需求侧响应的行为决策建模与分析[J]. 综合智慧能源, 2022, 44(2):80-88. doi: 10.3969/j.issn.2097-0706.2022.02.011
	JI Bing, SUN Hui, CHANG Li, et al. Modeling and analysis on decision making behavior of loyal users participating in demand-side response[J]. Intrgrated Intelligent Energy, 2022, 44(2):80-88.
[5]	闪鑫, 陆晓, 翟明玉, 等. 人工智能应用于电网调控的关键技术分析[J]. 电力系统自动化, 2019, 43(1):49-57.
	SHAN Xin, LU Xiao, ZHAI Mingyu, et al. Analysis of key technologies for artificial intelligence applied to power grid dispatch and control[J]. Automation of Electric Power Systems, 2019, 43(1):49-57.
[6]	LIU Y, HUANG X, LI S, et al. A construction method of power grid monitoring knowledge graph[J]. Journal of Physics:Conference Series, 2022, 2166(1):12-14.
[7]	赵俊峰, 庄哲寅, 承轶青, 等. 资源描述框架语义网视角下的智能电网模型[J]. 华电技术, 2014, 36(4):19-21.
	ZHAO Junfeng, ZHUANG Zheyin, CHENG Yiqing, et al. Intelligent power grid model under visual angle of RDF Semantic Web[J]. Huadian Technology, 2014, 36(4): 19-21.
[8]	CHEN Z, MI W, LIN J, et al. Discussion on intelligence assistant scheme of dispatching and control operation in power grid[J]. Automation of Electric Power Systems, 2019, 43(22):173-178.
[9]	吴俊勇. 国内外智能电网的发展战略[J]. 变频器世界, 2011, 34(9):36-37.
	WU Junyong. Development strategy of smart grid at China and abroad[J]. The World of Inverters, 2011, 34(9):36-37.
[10]	巫飞新. 国内外智能电网技术发展现状[J]. 电气开关, 2012, 50(2):3-6.
	WU Feixin. Development status of smart grid technology home and abroad[J]. Electric Switchgear, 2012, 50(2):3-6.
[11]	史梦, 张志生, 罗学礼, 曹敏. 智能电网建设思路及国内外发展战略[C]// 2010年云南电力技术论坛论文集(文摘部分), 2010:1571-1575.
[12]	FANG J, WANG Y, LEI Z, et al. Control Strategy and Performance Analysis of Electrochemical Energy Storage Station Participating in Power System Frequency Regulation:A case study of the jiangsu power grid[J]. Sustainability, 2022, 14(15):9189. doi: 10.3390/su14159189
[13]	HUANG Y, LI H, WANG Z, et al. Research on the mechanism and forecast opower grid regulation policy under the background of new electricity reform in China[J]. Journal of Physics:Conference Series, 2019, 1176(4):2030.
[14]	李明节, 陶洪铸, 许洪强, 等. 电网调控领域人工智能技术框架与应用展[J]. 电网技术, 2020, 44(2):393-400.
	LI Mingjie, TAO Hongzhu, XU Hongqiang, et al. The technical framework and application prospect of artificial intelligence application in the field of power grid dispatching and control[J]. Power System Technology, 2020, 44(2):393-400.
[15]	ELFAKI, ABDELRAHMAN O, SIM L, et al. Designing learning object repository using first order logic[C]// 2nd IIAI International Conference on Advanced Applied Informatics,IIAI-AAI 2013,413-414, 2013.
[16]	范士雄, 李立新, 王松岩, 等. 人工智能技术在电网调控中的应用研究[J]. 电网技术, 2020, 44(2):401-411.
	FAN Shixiong, LI Lixin, WANG Songyan, et al. Bowen.application analysis and exploration of artificial intelligence technology in power grid dispatch and control[J]. Power System Technology, 2020, 44(2):401-411.
[17]	CLAESSENS B, VRANCX P, RUELENS F. Convolutional neural networks for automatic state-time feature extraction in reinforcement learning applied to residential load control[J], IEEE Transactions on Smart Grid, 2016, 9(4):3259-3269. doi: 10.1109/TSG.2016.2629450
[18]	李志. 智能电网调度控制系统的运行安全策略[J]. 集成电路应用, 2022, 39(11):132-133.
	LI Zhi. Strategy of operation security of smart grid dispatching control system[J]. Application of IC, 2022, 39(11):132-133.
[19]	LIU S, GAO Y, YANG H, et al. Optimal guidance strategy for flexible load based on hybrid direct load control and time of use[J]. Global Energy Interconnection, 2023, 6(3):297-307. doi: 10.1016/j.gloei.2023.06.004
[20]	AISLING P, CONSTANCE C, KYRI B, et al. GridLearn:Multiagent reinforcement learning for grid-aware building energy management[J]. Electric Power Systems Research, 2022,213.
[21]	华咏竹, 谢强强, 秦会斌, 等. 计及用户端调节容量的变频空调自适应电压调控策略[J]. 综合智慧能源, 2022, 44(2):21-28. doi: 10.3969/j.issn.2097-0706.2022.02.004
	HUA Yongzhu, XIE Qiangqiang, QING Huibin, et al. Adaptive voltage regulation strategy for inverter air conditioners considering the regulation capacity on user side[J]. Intrgrated Intelligent Energy, 2022, 44(2):21-28.
[22]	童家麟, 洪庆, 吕洪坤, 等. 电源侧储能技术发展现状及应用前景综述[J]. 华电技术, 2021, 43(7):17-23.
	TONG Jialin, HONG Qing, LYU Hongkun, et al. Development status and application prospect of power side energy storage technology[J]. Huadian Technology, 2021, 43(7):17-23.
[23]	赵建立, 汤卓凡, 王桂林, 等. 具有储能作用的用户侧资源运行特性[J]. 综合智慧能源, 2022, 44(2):8-14. doi: 10.3969/j.issn.2097-0706.2022.02.002
	ZHAO Jianli, TANG Zhuofan, WANG Guilin, et al. Operation characteristics of user-side resources with energy storage function[J]. Intrgrated Intelligent Energy, 2022, 44(2):8-14.
[24]	LEON T, ALEXANDER S, FLORIAN S, et al. Pandapower an open-source python tool for convenient modeling, analysis,and optimization of electric power systems[J]. IEEE Transactions on Power Systems, 2018, 33(6):6510-6521. doi: 10.1109/TPWRS.59
[25]	DOLATABADI S, MAEDEH G, PIERLUIGI S, et al. An enhanced IEEE 33 bus benchmark test system for distribution system studies[J]. IEEE Transactions on Power Systems, 2021, 36(3):2565-2572. doi: 10.1109/TPWRS.2020.3038030
[26]	BUŞONIU L, BABUŠKA R, BART D. A comprehensive survey of multiagent reinforcement learning[J]. IEEE Transactions on Systems,Man,and Cybernetics, 2008, 310(2):156-172.
[27]	TAN M. Multi-agent reinforcement learning: Independent vs.cooperative agents[C]// Proceedings of the Tenth International Conference on Machine Learning,ICML 1993,330-337
[28]	WILLIAMS J. Simple statistical gradient-following algorithms for connectionist reinforcement learning[J]. Machine Learning, 1992, 8(3):229-256.
[29]	谢昕怡, 应黎明, 田书圣, 等. 基于MADDPG和智能合约的微电网交易决策优化[J]. 电力建设, 2022, 43(11):142-150. doi: 10.12204/j.issn.1000-7229.2022.11.014
	XIE Xingyi, YING Liming, TIAN Shushen, et al. Optimization of microgrid trading strategy based on maddpg and smart contracts[J]. Electric Power Construction, 2022, 43(11):142-150. doi: 10.12204/j.issn.1000-7229.2022.11.014
[30]	DAVID B, ZHANG X, DYLAN W, et al. Power gridworld: A framework for multi-agentreinforcement learning in power systems[C]// Proceedings of the 2022 13th ACM International Conference on Future Energy Systems,Energy, 2022,565-570.
[31]	孟安波, 王鹏, 丁伟锋, 等. 基于强化学习及纵横交叉粒子群算法的电网最优潮流计算[J]. 华电技术, 2021, 43(8):74-82.
	MENG Anbo, WANG Peng, DING Weifeng, et al. Optimal power flow calculation of power grid based on reinforcement learning and crisscross PSO algorithm particle swarm optimization[J]. Huadian Technology, 2021, 43(8):74-82.

建筑类型	具体建筑用途	当前类型数量
1	中型写字楼	4
2	快餐厅	37
3	零售商店	5
4	购物商店	1
5,6,7,8,9	多户住宅	145

可控单元	参数	值
光伏系统	太阳能板容量/kW	40
热水供应系统	热泵性能系数	0.22
	冷/热储水罐容量（标准单位）	1.5/0
	冷/热储水罐热损失系数	0.006/0.008
	加热/降温目标度/℃	45/9
	电热水器效率	0.92
储能系统	电池容量/（kW·h）	75
	电池效率	0.9
	额定功率/kW	30
	功率效率曲线	[0，0.83][0.3，0.85]
	容量功率曲线	[0.0，1][0.8，0.95]

参数	默认值
经验回放缓冲区容量	750
经验抽取数	32
折扣因子	0.95
目标网络更新权值	0.01
actor网络学习率	0.001
critic网络学习率	0.001
训练前数据收集轮次	3
动态学习率调整系数	0.3

项目	调控前	调控后
0.99~1.01电压比例/%	35.458	46.557
平均值	1.003 14	1.003 03
样本方差	0.000 360 08	0.000 337 17

基于MADDPG算法的建筑群柔性负荷优化调控方法

Optimized control method for flexible load of a building complex based on MADDPG reinforcement learning

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 12

参考文献 31

相关文章 15

编辑推荐

Metrics

本文评价

[1]	王永真, 韩艺博, 韩恺, 韩俊涛, 宋阔, 张兰兰. 基于知识图谱的数据中心综合能源系统研究综述[J]. 综合智慧能源, 2023, 45(7): 1-10.
[2]	胡泽, 朱子晴, 卜思齐, 陈家荣, 魏翔. 基于深度强化学习的区域综合能源定价策略研究[J]. 综合智慧能源, 2023, 45(7): 87-96.
[3]	窦真兰, 沈建忠, 张春雁, 江晶晶, 陈祺, 陈婧. 考虑供需不确定性的区域综合能源系统时间解耦分层能量管理[J]. 综合智慧能源, 2023, 45(6): 17-24.
[4]	王鑫, 陈祖翠, 卞在平, 王业耀, 吴育苗. 基于粒子群优化算法的智慧微电网风光储容量优化配置[J]. 综合智慧能源, 2022, 44(6): 52-58.
[5]	郭恒元, 冯小峰, 李国栋, 段志国, 李远征. 含制氢装置的机组组合与检修低碳协同优化研究[J]. 综合智慧能源, 2022, 44(5): 78-87.
[6]	刘静, 史梦鸽, 胡永锋. 含电池储能系统的智能楼宇多阶段能量管理策略[J]. 综合智慧能源, 2022, 44(3): 29-37.
[7]	夏雪, 张瑞曦, 王磊, 丁燃, 湛艳. 面向配电网数智化转型的高级配电管理系统研究与展望[J]. 综合智慧能源, 2022, 44(12): 33-39.
[8]	侯鲁洋, 葛磊蛟, 王飚, 王宣元, 徐连明, 王莉. 面向新型产消者的综合能源系统和电力市场研究[J]. 综合智慧能源, 2022, 44(12): 40-48.
[9]	敦文斌, 陈业, 王大伟, 马晓慧. 零碳视角下电气化码头源网荷储协同优化研究[J]. 综合智慧能源, 2022, 44(12): 68-74.
[10]	许志荣, 张高瑞. 响应用户有限理性需求的微电网优化策略[J]. 综合智慧能源, 2022, 44(11): 43-49.
[11]	杨梅, 周喜超, 魏强, 梁大伟, 吴茂乾. 考虑经济效益目标的源网荷储综合能源系统动态最优能流优化研究[J]. 综合智慧能源, 2022, 44(11): 63-69.
[12]	秦羽飞, 葛磊蛟, 王波. 能源互联网群体智能协同控制与优化技术[J]. 华电技术, 2021, 43(9): 1-13.
[13]	刘金豆, 成杰, 俞高伟. 基于低压直流配电网并网的并离网一体光储发电系统研究[J]. 华电技术, 2021, 43(4): 63-70.
[14]	冯力勇，张云. 考虑电池能效的电网侧电化学储能电站最优功率控制策略研究[J]. 华电技术, 2020, 42(4): 37-41.
[15]	孙书刚，朱昱，钱兵，李小武，倪红军. 基于遗传算法的涂层转接工艺优化[J]. 华电技术, 2020, 42(2): 72-75.