Pricing strategy in district-level integrated energy market based on deep reinforcement learning

doi:10.3969/j.issn.2097-0706.2023.07.010

Abstract

Abstract:

Integrated energy market （IEM）， being able to integrate multiple forms of energy transactions and promote the efficient use of energy， is growing and gradually taking place the traditional energy markets. District integrated energy market （DIEM）， which serves as a link between the supply and demand side， is crucial for energy transaction and pricing， and affects the operation of integrated energy systems. Given this context， a DIEM transaction structure is constructed to optimize the pricing strategy for Energy Service Providers （IESPs） and the demand response mechanism for Integrated Energy Consumers （IECs）. The double-layer decision-making optimization takes into account the elasticity of the energy demand， the uncertainty of the output of renewable energy sources， and privacy protection comprehensively. The optimal pricing of the IESP can be obtained by Deep Deterministic Policy Gradient （DDPG），which is compared with the pricing strategy made by Deep-Q-Learning（DQN） in a simulation case. The simulation analyzes the coupling relationship of energy prices in DIEM and the interaction between integrated energy pricing strategy and demand elasticity， showing that the revenue of the Integrated energy system obtained by DDPG is 6.8% higher than that made based on DQN.

Key words: integrated energy system, district integrated energy market, pricing strategy, demand response, deep reinforcement learning

CLC Number:

TK01

HU Ze, ZHU Ziqing, BU Siqi, CHAN Jiarong, WEI Xiang. Pricing strategy in district-level integrated energy market based on deep reinforcement learning[J]. Integrated Intelligent Energy, 2023, 45(7): 87-96.

Figures/Tables 14

Fig.1

Fig.2

Fig.3

Fig.4

Table 1

Parameters settings of the IESP

变量	参数	变量	参数
$η C H P, e$	0.3	$η C H P, h$	0.25
$λ g a s$ /(kW·h·m^-3)	9.97	$w i n c$ /(m·s^-1)	3
$w r a t e d c$ /(m·s^-1)	12	$w o u t c$ /(m·s^-1)	22
$S p v n$ /m²	500	$η p v n$	0.15

Table 1

Fig.5

Fig.6

Fig.7

Table 2

Table 3

Fig.8

Fig.9

Fig.10

Fig.11

References 21

[1]	董朝阳, 赵俊华, 文福拴, 等. 从智能电网到能源互联网:基本概念与研究框架[J]. 电力系统自动化, 2014, 38(15):1-11.
	DONG Zhaoyang, ZHAO Junhua, WEN Fushuan, et al. From smart grid to energy internet:Basic concept and research framework[J]. Automation of Electric Power System, 2014. 38(15):1-11.
[2]	谈金晶, 李扬. 多能源协同的交易模式研究综述[J]. 中国电机工程学报, 2019, 39(22):6483-6497.
	TAN Jinjing, LI Yang. Review on transaction mode in multi-energy collaborative market[J]. Proceedings of the CSEE, 2019, 39(22):6483-6497.
[3]	吕佳炜, 张沈习, 程浩忠, 等. 考虑互联互动的区域综合能源系统规划研究综述[J]. 中国电机工程学报, 2021, 41(12):4001-4021.
	LYU Jiawei, ZHANG Shenxi, CHENG Haozhong, et al. Review on district-level integrated energy system planning considering interconnection and interaction[J]. Proceedings of the CSEE, 2021, 41(12):4001-4021.
[4]	张颖梓, 李华强, 李旭翔, 等. 基于用户需求行为的综合能源服务产品定价策略研究[J]. 电力系统保护与控制, 2021, 49(17):121-129.
	ZHANG Yingzi, LI Huaqiang, LI Xuxiang, et al. Pricing strategy of integrated energy service products based on user demand behavior[J]. Power System Protection and Control, 2021, 49(17):121-129.
[5]	王俊, 徐箭, 柯德平, 等. 考虑多市场主体参与的气电区域综合能源系统市场定价策略[J]. 电力自动化设备, 2022, 42(9):18-26.
	WANG Jun, XU Jian, KE Deping, et al. Market pricing strategy for gas-electricity integrated energy system considering multiple market entities[J]. Electric Power Automation Equipment, 2022, 42(9):18-26.
[6]	卞心怡, 范宏, 曾博. 计及多重不确定性的低碳综合能源系统动态定价策略[J/OL]. 电测与仪表,1-12(2023-04-10)[2023-05-05]. http://kns.cnki.net/kcms/detail/23.1202.TH.20230407.1522.004.html.
	BIAN Xinyi, FAN Hong, ZENG Bo. Dynamic pricing strategy of low-carbon integrated energy system considering multiple uncertainties[J/OL]. Electrical Measurement & Instrumentation,1-12(2023-04-10)[2023-05-05]. http://kns.cnki.net/kcms/detail/23.1202.TH.20230407.1522.004.html.
[7]	曹阳, 喻洁, 李扬, 等. 基于演化博弈的区域综合能源市场零售侧竞争策略选择方法[J]. 电力系统自动化, 2023, 47(5):104-113.
	CAO Yang, YU Jie, LI Yang, et al. Evolutionary game based competition strategy selection method for retail side of regional integrated energy market[J]. Automation of Power Electric System, 2023, 47(5):104-113.
[8]	DOU X, WANG J, HU Q, et al. Bi-level bidding and multi-energy retail packages for integrated energy service providers considering multi-energy demand elasticity[J]. CSEE Journal of Power and Energy Systems, 2020.
[9]	杨挺, 赵黎媛, 刘亚闯, 等. 基于深度强化学习的综合能源系统动态经济调度[J]. 电力系统自动化, 2021, 45(5):39-47.
	YANG Ting, ZHAO Liyuan, LIU Yachuang, et al. Dynamic economic dispatch for integrated energy system based on deep reinforcement learning[J]. Automation of Power Electric System, 2021, 45(5):39-47.
[10]	陈传杰, 杨海柱, 李梦龙, 等. 基于实时定价机制的综合能源系统多时间尺度优化调度[J]. 科学技术与工程, 2021, 21(12): 4968-4974.
	CHEN Chuanjie, YANG Haizhu, LI Menglong, et al. Multi-time scale optimal scheduling of integrated energy systems based on real-time pricing mechanism[J]. Science Technology and Engineering, 2021, 21(12): 4968-4974.
[11]	冯斌, 胡轶婕, 黄刚, 等. 基于深度强化学习的新型电力系统调度优化方法综述[J/OL]. 电力系统自动化:1-22(2023-04-03)[2023-05-07]. http://kns.cnki.net/kcms/detail/32.1180.TP.20230331.1354.004.html.
	FENG Bin, HU Yijie, HUANG Gang, et al. Review on optimization methods for new power system dispatch based on deep reinforcement learning[J/OL]. Automation of Power Electric System:1-22(2023-04-03)[2023-05-07]. http://kns.cnki.net/kcms/detail/32.1180.TP.20230331.1354.004.html.
[12]	SILVER D, HUANG A, MADDISON C J, et al. Mastering the game of Go with deep neural networks and tree search[J]. Nature, 2016, 529(7587): 484-489. doi: 10.1038/nature16961
[13]	MNIH V, KAVUKCUOGLU K, SILVER D, et al. Playing atari with deep reinforcement learning[J]. arXiv preprint arXiv:1312.5602, 2013.
[14]	孙庆凯, 王小君, 王怡, 等. 基于多智能体Nash-Q强化学习的综合能源市场交易优化决策[J]. 电力系统自动化, 2021, 45(16):124-133.
	SUN Qingkai, WANG Xiaojun, WANG Yi, et al. Optimal trading decision-making for integrated energy market based on multi-agent Nash-Q reinforcement learning[J]. Automation of Power Electric System, 2021, 45(16):124-133.
[15]	WU X, LIU B, YUAN S, et al. Pricing strategy of regional integrated energy system considering privacy protection based on deep reinforcement learning[J]. CSEE Journal of Power and Energy Systems, 2022.
[16]	FAZLALIPOUR P, EHSAN M, MOHAMMADI-IVATLOO B. Risk-aware stochastic bidding strategy of renewable micro-grids in day-ahead and real-time markets[J]. Energy, 2019, 171: 689-700. doi: 10.1016/j.energy.2018.12.173
[17]	LILLICRAP T P, HUNT J J, PRITZEL A, et al. Continuous control with deep reinforcement learning[J]. arXiv preprint arXiv:1509.02971, 2015.
[18]	BOUSSETTA M, MOTAHHIR S, BACHTIRI R E, et al. Design and embedded implementation of a power management controller for wind-PV-diesel microgrid system[J]. International Journal of Photoenergy, 2019: 1-16.
[19]	Information on individual ISO markets, market products, and ancillary services[EB/OL].[2023-06-03]. https://www.iso-ne.com/markets-operations/markets/.
[20]	Heat demand profile[EB/OL].[2023-06-03]. https://www.esru.strath.ac.uk/EandE/Web_sites/17-18/gigha/heat-demand-profile.html.
[21]	Decision intelligence,across the enterprise[EB/OL].[2023-06-03]. https://www.gurobi.com/.

神经网络	动作Actor	评价Critic
隐藏层数量	2	2
神经元数量	[64,32]	[64,32]
激活函数	tanh	ReLU
学习率	1e^-4	1e^-4
软更新参数	1e^-3	1e^-3
延迟更新频率	1	2
优化求解器	Adam optimizer	Adam optimizer

训练参数	参数值
经验回放缓冲区规模	1e⁶
经验回放起始数	100
回放经验批量大小	16
折扣率	0.99