Optimal scheduling of intelligent virtual power plants based on explainable reinforcement learning

doi:10.3969/j.issn.2097-0706.2025.01.001

Abstract

Abstract:

With the increasing popularity of electric vehicles（EVs），energy systems are becoming more complex. Virtual power plants（VPPs）can aggregate and optimize distributed energy resources such as distributed generation，energy storage systems，controllable loads，and EVs through internet of things（IoT）and artificial intelligence（AI）technologies，enhancing energy efficiency and facilitating the consumption of non-renewable energy while reinforcing grid stability. However，current AI technologies lack reliability and transparency in high-safety applications like power systems，potentially making it challenging for users and operators to understand how algorithms make specific energy allocation decisions. To address the balance between achieving optimal scheduling of VPPs utilizing AI and explaining the decision-making processes，this study proposed an interactive framework based on explainable reinforcement learning. This framework employed the proximal policy optimization（PPO）algorithm for optimal scheduling of VPPs and constructed an explainable reinforcement learning framework using decision trees to provide transparent decision support that enabled non-expert users to understand AI's decision-making processes in regulating energy systems. The results indicated that compared to traditional reinforcement learning optimization methods，this approach not only improved energy allocation efficiency but also strengthened user trust in intelligent VPP management systems by enhancing model interpretability.

Key words: virtual power plant, electric vehicle, proximal policy optimization algorithm, reinforcement learning, decision tree, explainable framework, distributed energy, artificial intelligence

CLC Number:

TK019

YUAN Xiaoke, SHEN Shilan, ZHANG Maosong, SHI Chenxu, YANG Lingxiao. Optimal scheduling of intelligent virtual power plants based on explainable reinforcement learning[J]. Integrated Intelligent Energy, 2025, 47(1): 1-9.

Figures/Tables 9

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Fig.8

Fig.9

References 26

[1]	刘媛媛, 刘芳芳, 贾天翔, 等. 商住园区综合能源供暖(冷)系统的方案设计及运行经济性研究[J]. 综合智慧能源, 2023, 45(12): 20-28. doi: 10.3969/j.issn.2097-0706.2023.12.003
	LIU Yuanyuan, LIU Fangfang, JIA Tianxiang, et al. Design of the integrated energy heating (cooling) system for a commercial and residential park and its economy analysis[J]. Integrated Intelligent Energy, 2023, 45(12): 20-28. doi: 10.3969/j.issn.2097-0706.2023.12.003
[2]	田泽禹, 沙钊旸, 赵全斌, 等. 针对温控负载变化的虚拟电厂控制策略研究[J]. 综合智慧能源, 2024, 46(1): 28-37. doi: 10.3969/j.issn.2097-0706.2024.01.004
	TIAN Zeyu, SHA Zhaoyang, ZHAO Quanbin, et al. Research on control strategy for virtual power plants in response to thermostatically controlled loads[J]. Integrated Intelligent Energy, 2024, 46(1): 28-37. doi: 10.3969/j.issn.2097-0706.2024.01.004
[3]	孙秋野, 姚葭, 王一帆. 从虚拟电厂到真实电量: 虚拟电厂研究综述与展望[J]. 发电技术, 2023, 4(5): 583-601.
	SUN Qiuye, YAO Jia, WANG Yifan. From virtual power plant to real electricity:Summary and prospect of virtual power plant research[J]. Power Generation Technology, 2023, 44(5): 583-601. doi: 10.12096/j.2096-4528.pgt.23102
[4]	葛晓琳, 郝广东, 夏澍, 等. 考虑规模化电动汽车与风电接入的随机解耦协同调度[J]. 电力系统自动化, 2020, 44(4): 54-62.
	GE Xiaolin, HAO Guangdong, XIA Shu, et al. Stochastic decoupling collaborative dispatch considering integration of large-scale electric vehicles and wind power[J]. Automation of Electric Power Systems, 2020, 44(4): 54-62.
[5]	冶兆年, 赵长禄, 王永真, 等. 基于纳什议价的共享储能能源互联网络双目标优化[J]. 综合智慧能源, 2022, 44(7):40-48. doi: 10.3969/j.issn.2097-0706.2022.07.005
	YE Zhaonian, ZHAO Changlu, WANG Yongzhen, et al. Dua-objective optimization of energy networks with shared energy storage based on Nash bargaining[J]. Integrated Intelligent Energy, 2022, 44(7): 40-48. doi: 10.3969/j.issn.2097-0706.2022.07.005
[6]	刘建行, 刘方. 基于深度强化学习的梯级水蓄风光互补系统优化调度策略研究[J]. 广东电力, 2024, 37(5): 10-22.
	LIU Jianhang, LIU Fang. Research on optimized dispatching strategy of cascade hydropower-pumping-storage-wind-photovoltaic multi-energy complementary system based on deep reinforcement learning[J]. Guangdong Electric Power, 2024, 37(5): 10-22.
[7]	林彦旭, 高辉. 基于SSA-VMD-BiLSTM模型的充电站负荷预测方法[J]. 广东电力, 2024, 37(6): 53-61.
	LIN Yanxu, GAO Hui. Load prediction method of charging station based on SSA-VMD-BiLSTM model[J]. Guangdong Electric Power, 2024, 37(6): 53-61.
[8]	DA SILVA F L, NISHIDA C E H, ROIJERS D M, et al. Coordination of electric vehicle charging through multiagent reinforcement learning[J]. IEEE Transactions on Smart Grid, 2020, 11(3): 2347-2356.
[9]	PAN X T, WANG L P, QIU Q C, et al. Many-objective optimization for large-scale EVs charging and discharging schedules considering travel convenience[J]. Applied Intelligence, 2022, 52(3): 2599-2620.
[10]	ZHANG F Y, YANG Q Y, AN D. CDDPG:A deep-reinforcement-learning-based approach for electric vehicle charging control[J]. IEEE Internet of Things Journal, 2021, 8(5): 3075-3087.
[11]	LI H, LI G J, LIE T T, et al. Constrained large-scale real-time EV scheduling based on recurrent deep reinforcement learning[J]. International Journal of Electrical Power & Energy Systems, 2023, 144: 108603.
[12]	杨挺, 赵黎媛, 刘亚闯, 等. 基于深度强化学习的综合能源系统动态经济调度[J]. 电力系统自动化, 2021, 45(5): 39-47.
	YANG Ting, ZHAO Liyuan, LIU Yachuang, et al. Dynamic economic dispatch for integrated energy system based on deep reinforcement learning[J]. Automation of Electric Power Systems, 2021, 45(5): 39-47.
[13]	杨志学, 任洲洋, 孙志媛, 等. 基于近端策略优化算法的新能源电力系统安全约束经济调度方法[J]. 电网技术, 2023, 47(3): 988-998.
	YANG Zhixue, REN Zhouyang, SUN Zhiyuan, et al. Security-constrained economic dispatch of renewable energy integrated power systems based on proximal policy optimization algorithm[J]. Power System Technology, 2023, 47(3): 988-998.
[14]	WAN Z Q, LI H P, HE H B, et al. Model-free real-time EV charging scheduling based on deep reinforcement learning[J]. IEEE Transactions on Smart Grid, 2019, 10(5): 5246-5257.
[15]	LI H P, WAN Z Q, HE H B. Constrained EV charging scheduling based on safe deep reinforcement learning[J]. IEEE Transactions on Smart Grid, 2020, 11(3): 2427-2439.
[16]	黎静华, 朱梦姝, 陆悦江, 等. 综合能源系统优化调度综述[J]. 电网技术, 2021, 45(6): 2256-2272.
	LI Jinghua, ZHU Mengshu, LU Yuejiang, et al. Review on optimal scheduling of integrated energy systems[J]. Power System Technology, 2021, 45(6): 2256-2272.
[17]	蔺伟山, 王小君, 孙庆凯, 等. 计及安全约束的综合能源系统深度强化学习优化调度策略研究[J]. 电网技术, 2023, 47(5): 1970-1983.
	LIN Weishan, WANG Xiaojun, SUN Qingkai, et al. Optimal dispatch strategy of integrated energy system based on deep reinforcement learning considering security constraints[J]. Power System Technology, 2023, 47(5): 1970-1983.
[18]	李佳玮. 基于图神经网络的配电网故障定位方法[D]. 北京: 北京交通大学, 2022.
	LI Jiawei. Fault location method of distribution network based on graph neural network[D]. Beijing: Beijing Jiaotong University, 2022.
[19]	崔杨, 王议坚, 黄彦浩, 等. 基于多元注意力框架与引导式监督学习的闭环风电功率超短期预测策略[J]. 中国电机工程学报, 2023, 43(4): 1334-1347.
	CUI Yang, WANG Yijian, HUANG Yanhao, et al. Closed-loop wind power ultra-short-term forecasting strategy based on multi-attention framework and guided supervised learning[J]. Proceedings of the CSEE, 2023, 43(4): 1334-1347.
[20]	HEO S, KO J, KIM S, et al. Explainable AI-driven net-zero carbon roadmap for petrochemical industry considering stochastic scenarios of remotely sensed offshore wind energy[J]. Journal of Cleaner Production, 2022, 379: 134793.
[21]	孙庆凯, 王小君, 张义志, 等. 基于LSTM和多任务学习的综合能源系统多元负荷预测[J]. 电力系统自动化, 2021, 45(5): 63-70.
	SUN Qingkai, WANG Xiaojun, ZHANG Yizhi, et al. Multiple load prediction of integrated energy system based on long short-term memory and multi-task learning[J]. Automation of Electric Power Systems, 2021, 45(5): 63-70.
[22]	周挺, 杨军, 詹祥澎, 等. 一种数据驱动的暂态电压稳定评估方法及其可解释性研究[J]. 电网技术, 2021, 45(11): 4416-4425.
	ZHOU Ting, YANG Jun, ZHAN Xiangpeng, et al. Data-driven method and interpretability analysis for transient voltage stability assessment[J]. Power System Technology, 2021, 45(11): 4416-4425.
[23]	龙寰, 杨婷, 徐劭辉, 等. 基于数据驱动的风电机组状态监测与故障诊断技术综述[J]. 电力系统自动化, 2023, 47(23): 55-69.
	LONG Huan, YANG Ting, XU Shaohui, et al. Review of data-driven condition monitoring and fault diagnosis technologies for wind turbines[J]. Automation of Electric Power Systems, 2023, 47(23): 55-69.
[24]	苏向敬, 周汶鑫, 李超杰, 等. 基于双重注意力LSTM神经网络的可解释海上风电出力预测[J]. 电力系统自动化, 2022, 46(7): 141-151.
	SU Xiangjing, ZHOU Wenxin, LI Chaojie, et al. Interpretable offshore wind power output forecasting based on long short-term memory neural network with dual-stage attention[J]. Automation of Electric Power Systems, 2022, 46(7): 141-151.
[25]	ZHANG K, ZHANG J, XU P D, et al. Explainable AI in deep reinforcement learning models for power system emergency control[J]. IEEE Transactions on Computational Social Systems, 2022, 9(2): 419-427.
[26]	ZHANG K, ZHANG J, XU P D, et al. A multi-hierarchical interpretable method for DRL-based dispatching control in power systems[J]. International Journal of Electrical Power & Energy Systems, 2023, 152: 109240.