Optimized control method for flexible load of a building complex based on MADDPG reinforcement learning

doi:10.3969/j.issn.2097-0706.2023.07.007

Abstract

Abstract:

The power grid dispatch environment and information organization environment have become more complex， and the difficulty of power grid regulation has gradually increased. Since deep reinforcement learning technology is of effective perception on complex system operation statuses，strong adaptability and good scalability，a distribution network optimization scheduling method based on deep reinforcement learning is proposed. Based on the simulated source-network-load-storage integrated distribution network model of a building complex，Multi-Agent Deep Deterministic Policy Gradient（MADDPG）algorithm was statically optimized from its principle.The model and real data were input into a multi-agent reinforcement learning framework suitable for grid-level objectives，and the optimized algorithm was tried to regulate the voltage of the distribution network system. The results show that the algorithm basically eliminates the abnormal peak voltages and reduces the overall voltage deviation.The optimized multi-objective oriented algorithm reduces the load-generated power difference while levelling the voltage off at a low level. The optimized control method for building complex flexible load based on reinforcement learning is proven to be effective.

Key words: microgrid regulation, energy management, deep reinforcement learning, deterministic policy gradient, multi-objective optimization, source-grid-load- storage, flexible load of buildings

CLC Number:

TK01⁺8：TU18

BAO Yixin, XU Luoyun, YANG Qiang. Optimized control method for flexible load of a building complex based on MADDPG reinforcement learning[J]. Integrated Intelligent Energy, 2023, 45(7): 61-69.

Figures/Tables 12

Table 1

Fig.1

Fig.2

Table 2

Table 3

Fig.3

Fig.4

Fig.5

Table 4

Fig.6

Fig.7

Fig.8

References 31

[1]	王晓燕, 宋方宇轩, 卢珊. 电网调控中人工智能应用的关键技术研究[J]. 科技与创新, 2022,(23):4-6,11
	WANG Xiaoyan, SONG Fangyuxuan, LU Shan. Research on key technologies of artificial intelligence application in power grid regulation[J]. Science and Technology & Innovation, 2022,(23):4-6,11.
[2]	崔文虎. 电力系统故障演化建模与分析[D]. 成都: 中国电子科技大学, 2018.
	CUI Wenhu. Power system fault evolution modeling and analysis[D]. Chengdu: University of Electronic Science and Technology of China, 2018.
[3]	姜丽珍, 董淑杰, 闫振伟, 等. 电网调控技术在电力系统中的应用[J]. 电子制作, 2019(12):69-70.
	JIANG Lizhen, DONG Shujie, YAN Zhenwei, et al. Application of power grid regulation technology in power system[J]. Practical Electronics, 2019(12):69-70.
[4]	吉斌, 孙绘, 昌力, 等. 黏性电力用户参与需求侧响应的行为决策建模与分析[J]. 综合智慧能源, 2022, 44(2):80-88. doi: 10.3969/j.issn.2097-0706.2022.02.011
	JI Bing, SUN Hui, CHANG Li, et al. Modeling and analysis on decision making behavior of loyal users participating in demand-side response[J]. Intrgrated Intelligent Energy, 2022, 44(2):80-88.
[5]	闪鑫, 陆晓, 翟明玉, 等. 人工智能应用于电网调控的关键技术分析[J]. 电力系统自动化, 2019, 43(1):49-57.
	SHAN Xin, LU Xiao, ZHAI Mingyu, et al. Analysis of key technologies for artificial intelligence applied to power grid dispatch and control[J]. Automation of Electric Power Systems, 2019, 43(1):49-57.
[6]	LIU Y, HUANG X, LI S, et al. A construction method of power grid monitoring knowledge graph[J]. Journal of Physics:Conference Series, 2022, 2166(1):12-14.
[7]	赵俊峰, 庄哲寅, 承轶青, 等. 资源描述框架语义网视角下的智能电网模型[J]. 华电技术, 2014, 36(4):19-21.
	ZHAO Junfeng, ZHUANG Zheyin, CHENG Yiqing, et al. Intelligent power grid model under visual angle of RDF Semantic Web[J]. Huadian Technology, 2014, 36(4): 19-21.
[8]	CHEN Z, MI W, LIN J, et al. Discussion on intelligence assistant scheme of dispatching and control operation in power grid[J]. Automation of Electric Power Systems, 2019, 43(22):173-178.
[9]	吴俊勇. 国内外智能电网的发展战略[J]. 变频器世界, 2011, 34(9):36-37.
	WU Junyong. Development strategy of smart grid at China and abroad[J]. The World of Inverters, 2011, 34(9):36-37.
[10]	巫飞新. 国内外智能电网技术发展现状[J]. 电气开关, 2012, 50(2):3-6.
	WU Feixin. Development status of smart grid technology home and abroad[J]. Electric Switchgear, 2012, 50(2):3-6.
[11]	史梦, 张志生, 罗学礼, 曹敏. 智能电网建设思路及国内外发展战略[C]// 2010年云南电力技术论坛论文集(文摘部分), 2010:1571-1575.
[12]	FANG J, WANG Y, LEI Z, et al. Control Strategy and Performance Analysis of Electrochemical Energy Storage Station Participating in Power System Frequency Regulation:A case study of the jiangsu power grid[J]. Sustainability, 2022, 14(15):9189. doi: 10.3390/su14159189
[13]	HUANG Y, LI H, WANG Z, et al. Research on the mechanism and forecast opower grid regulation policy under the background of new electricity reform in China[J]. Journal of Physics:Conference Series, 2019, 1176(4):2030.
[14]	李明节, 陶洪铸, 许洪强, 等. 电网调控领域人工智能技术框架与应用展[J]. 电网技术, 2020, 44(2):393-400.
	LI Mingjie, TAO Hongzhu, XU Hongqiang, et al. The technical framework and application prospect of artificial intelligence application in the field of power grid dispatching and control[J]. Power System Technology, 2020, 44(2):393-400.
[15]	ELFAKI, ABDELRAHMAN O, SIM L, et al. Designing learning object repository using first order logic[C]// 2nd IIAI International Conference on Advanced Applied Informatics,IIAI-AAI 2013,413-414, 2013.
[16]	范士雄, 李立新, 王松岩, 等. 人工智能技术在电网调控中的应用研究[J]. 电网技术, 2020, 44(2):401-411.
	FAN Shixiong, LI Lixin, WANG Songyan, et al. Bowen.application analysis and exploration of artificial intelligence technology in power grid dispatch and control[J]. Power System Technology, 2020, 44(2):401-411.
[17]	CLAESSENS B, VRANCX P, RUELENS F. Convolutional neural networks for automatic state-time feature extraction in reinforcement learning applied to residential load control[J], IEEE Transactions on Smart Grid, 2016, 9(4):3259-3269. doi: 10.1109/TSG.2016.2629450
[18]	李志. 智能电网调度控制系统的运行安全策略[J]. 集成电路应用, 2022, 39(11):132-133.
	LI Zhi. Strategy of operation security of smart grid dispatching control system[J]. Application of IC, 2022, 39(11):132-133.
[19]	LIU S, GAO Y, YANG H, et al. Optimal guidance strategy for flexible load based on hybrid direct load control and time of use[J]. Global Energy Interconnection, 2023, 6(3):297-307. doi: 10.1016/j.gloei.2023.06.004
[20]	AISLING P, CONSTANCE C, KYRI B, et al. GridLearn:Multiagent reinforcement learning for grid-aware building energy management[J]. Electric Power Systems Research, 2022,213.
[21]	华咏竹, 谢强强, 秦会斌, 等. 计及用户端调节容量的变频空调自适应电压调控策略[J]. 综合智慧能源, 2022, 44(2):21-28. doi: 10.3969/j.issn.2097-0706.2022.02.004
	HUA Yongzhu, XIE Qiangqiang, QING Huibin, et al. Adaptive voltage regulation strategy for inverter air conditioners considering the regulation capacity on user side[J]. Intrgrated Intelligent Energy, 2022, 44(2):21-28.
[22]	童家麟, 洪庆, 吕洪坤, 等. 电源侧储能技术发展现状及应用前景综述[J]. 华电技术, 2021, 43(7):17-23.
	TONG Jialin, HONG Qing, LYU Hongkun, et al. Development status and application prospect of power side energy storage technology[J]. Huadian Technology, 2021, 43(7):17-23.
[23]	赵建立, 汤卓凡, 王桂林, 等. 具有储能作用的用户侧资源运行特性[J]. 综合智慧能源, 2022, 44(2):8-14. doi: 10.3969/j.issn.2097-0706.2022.02.002
	ZHAO Jianli, TANG Zhuofan, WANG Guilin, et al. Operation characteristics of user-side resources with energy storage function[J]. Intrgrated Intelligent Energy, 2022, 44(2):8-14.
[24]	LEON T, ALEXANDER S, FLORIAN S, et al. Pandapower an open-source python tool for convenient modeling, analysis,and optimization of electric power systems[J]. IEEE Transactions on Power Systems, 2018, 33(6):6510-6521. doi: 10.1109/TPWRS.59
[25]	DOLATABADI S, MAEDEH G, PIERLUIGI S, et al. An enhanced IEEE 33 bus benchmark test system for distribution system studies[J]. IEEE Transactions on Power Systems, 2021, 36(3):2565-2572. doi: 10.1109/TPWRS.2020.3038030
[26]	BUŞONIU L, BABUŠKA R, BART D. A comprehensive survey of multiagent reinforcement learning[J]. IEEE Transactions on Systems,Man,and Cybernetics, 2008, 310(2):156-172.
[27]	TAN M. Multi-agent reinforcement learning: Independent vs.cooperative agents[C]// Proceedings of the Tenth International Conference on Machine Learning,ICML 1993,330-337
[28]	WILLIAMS J. Simple statistical gradient-following algorithms for connectionist reinforcement learning[J]. Machine Learning, 1992, 8(3):229-256.
[29]	谢昕怡, 应黎明, 田书圣, 等. 基于MADDPG和智能合约的微电网交易决策优化[J]. 电力建设, 2022, 43(11):142-150. doi: 10.12204/j.issn.1000-7229.2022.11.014
	XIE Xingyi, YING Liming, TIAN Shushen, et al. Optimization of microgrid trading strategy based on maddpg and smart contracts[J]. Electric Power Construction, 2022, 43(11):142-150. doi: 10.12204/j.issn.1000-7229.2022.11.014
[30]	DAVID B, ZHANG X, DYLAN W, et al. Power gridworld: A framework for multi-agentreinforcement learning in power systems[C]// Proceedings of the 2022 13th ACM International Conference on Future Energy Systems,Energy, 2022,565-570.
[31]	孟安波, 王鹏, 丁伟锋, 等. 基于强化学习及纵横交叉粒子群算法的电网最优潮流计算[J]. 华电技术, 2021, 43(8):74-82.
	MENG Anbo, WANG Peng, DING Weifeng, et al. Optimal power flow calculation of power grid based on reinforcement learning and crisscross PSO algorithm particle swarm optimization[J]. Huadian Technology, 2021, 43(8):74-82.

建筑类型	具体建筑用途	当前类型数量
1	中型写字楼	4
2	快餐厅	37
3	零售商店	5
4	购物商店	1
5,6,7,8,9	多户住宅	145

可控单元	参数	值
光伏系统	太阳能板容量/kW	40
热水供应系统	热泵性能系数	0.22
	冷/热储水罐容量（标准单位）	1.5/0
	冷/热储水罐热损失系数	0.006/0.008
	加热/降温目标度/℃	45/9
	电热水器效率	0.92
储能系统	电池容量/（kW·h）	75
	电池效率	0.9
	额定功率/kW	30
	功率效率曲线	[0，0.83][0.3，0.85]
	容量功率曲线	[0.0，1][0.8，0.95]

参数	默认值
经验回放缓冲区容量	750
经验抽取数	32
折扣因子	0.95
目标网络更新权值	0.01
actor网络学习率	0.001
critic网络学习率	0.001
训练前数据收集轮次	3
动态学习率调整系数	0.3

项目	调控前	调控后
0.99~1.01电压比例/%	35.458	46.557
平均值	1.003 14	1.003 03
样本方差	0.000 360 08	0.000 337 17