Strategic Investment Decision Making for Value Creation in Energy Transition: A Reinforcement Learning Approach

The gist The research develops a custom simulation environment and applies Reinforcement Learning to identify optimal investment policies for energy companies navigating the complex, uncertain transition toward net-zero emissions.

Strategic Framework and Objectives

The study conceptualizes the energy transition as a Sequential Decision-Making (SDM) process under uncertainty to evaluate different planning strategies The framework explores various decision strategies related to different portfolios for allocating funds across three sectors: oil & gas, renewables, and CO2 reduction This framework has three objectives: maximizing profit, minimizing CO2 social costs, and enhancing competitive advantage in the renewable energy sector Portfolio management involves the continuous or periodic process of reallocating funds across various financial investment opportunities to maximize returns while minimizing risks The way we approach portfolio management in the context of the energy transition is novel and takes a broader, more integrated perspective

Reinforcement Learning Methodology

The research utilizes Reinforcement Learning (RL) as a state-of-the-art approach to solve complex SDM problems within the defined framework The agent’s sequential decisions shape a virtual dynamic environment by influencing key variables such as oil and gas production, renewable energy output, CO2 emissions, and revenues Through repeated interaction, the RL algorithm explores the state space and learns an optimal policy under uncertainty The Deep Q-Network (DQN) is used to approximate the Q-function using deep neural networks to efficiently learn optimal policies in high dimensional state-action spaces The reward function combines three monetary objectives with penalties for regulatory constraints, such as net-zero emission regulations and smooth transition constraints The combined reward function for calculating reward in each interaction step, is defined as r i = wProfit ∗ NPV i + wSCC ∗ SCC i + wCA ∗ CA i + rNetZero + rsmoothness

Simulation Environment Dynamics

The simulation environment models a conceptual energy company in the energy transition over a 25-year period, from 2025 to 2050 The decision variables are the allocation of investment across the three sectors each year, expressed as fractions of the total available investment budget in that year and sum to one The environment incorporates stochastic variables such as oil and gas production, CO2 emission volumes, renewable energy production, energy prices, and operational expenses Oil and gas production is modeled based on investment level, reserve change, and production growth using equations describing the update process Renewable energy production is updated based on investment levels leading to additional capacity installation and electricity generation scaled by efficiency improvements CO2 emission modeling regresses on publicly available data from Equinor’s Climate Tables to calculate net annual CO2 emissions

Key Findings and Benchmark Comparison

The RL-based investment strategy dynamically reallocates funds over time, gradually reducing investments in oil and gas while increasing allocations to renewables and CO2 reduction Oil production peaks at about 109 million barrels of oil equivalent around 2033 and then declines to roughly 26 million by 2050, while renewable electricity generation expands to about 27,000 GWh over the same period The RL policy achieves near-zero CO2 emissions by 2050, with a consistent downward trend starting around 2033 Compared to several benchmark strategies that will be described in detail in the Results section, the RL policy achieves higher long-term value The strategy combined reward clearly outperforms others when integrating all three objectives When focusing only on maximizing profit, the strategy that allocates entirely to the oil and gas sector achieves the highest reward The results demonstrate that framing the energy transition as a multi-objective sequential decision making under uncertainty is essential for better investment decisions and value creation

Trade-off Analysis

Pareto analysis was conducted by training the DQN agent fifteen times with different objective weights to understand trade-offs between competing objectives Figure 10a shows a clear Pareto front: higher NPV accompanied by higher emissions in general The cheapest reductions therefore come first, and the last steps toward net-zero are the most expensive The results emphasize that the interactions among objectives are complex in a multi-objective setting and trade-off analysis must account for how all objectives interact Focusing on only two metrics may give an incomplete picture of the interactions

Decision Quality Assessment

The Decision Quality (DQ) chain was used to assess the quality of the proposed decision framework, with Creative Alternatives being identified as arguably the weakest link The framework is explicitly framed and includes clearly defined objectives converted into monetary units, which is a strong link in this study The problem is formulated as a Markov Decision Process and solved using a Deep Q-Network, which supports sound reasoning However, the transition from decision making to actual implementation remains an area requiring further work

The paper concludes that combining Sequential Decision-Making thinking with modern Reinforcement Learning techniques offers a powerful toolset for tackling the complex, uncertain challenges in energy transition investments. The proposed framework provides a flexible and accessible foundation for future research on decision-making in this high-stakes domain. The work highlights that achieving net-zero emissions by 2050 comes at a high cost of declining annual cash flow through the 2030s to a minimum in the early 2040s Such insight emphasizes the importance of advanced, long-term planning in navigating the energy transition with both strategic and financial foresight. The simulation environment can be made more realistic by incorporating additional critical variables such as energy demand to allow for the representation of supply-demand dynamics Future research can leverage historical data to build more context-specific stochastic models Future work can explore other multi-objective reinforcement learning approaches aimed at identifying Pareto-optimal policies The proposed framework is openly available to support reproducibility and benchmarking for future studies The simulation environment is publicly available at https://github.com/YasCheraghi/Energy-TransitionRL Third Paper PhD for research The paper demonstrates that the RL policy outperformed benchmark strategies, achieving the highest combined reward with a smooth change in allocation of investments over time The shift from oil and gas to renewables is gradual but accelerates after 2033, aligning with CO2 reduction targets while preserving financial stability The RL agent learns this behavior to take advantage of higher short-term returns in oil and gas and to generate capital for future investments in renewables and CCS This research provides a valuable tool for energy companies to better understand their values and the trade-offs between them

--- Page 1 ---

The gist The research develops a custom simulation environment and applies Reinforcement Learning to identify optimal investment policies for energy companies navigating the complex, uncertain transition toward net-zero emissions.

1. Bold Header: Deep Q-Network Architecture Enhancement

The DQN architecture should be adapted to handle vector-valued Q-functions, as Q-learning uses this difference [TD error], [which] also known as TD error, the first part of which (r + γ max a' Q(s', a')) acts as the “target” toward which the Q-value is adjusted for multiple objectives. This requires modifying the network output layer to predict a vector of values corresponding to each objective's reward component rather than a single scalar.

2. Bold Header: Multi-Objective Reward Function Refinement

The combined reward function needs refinement beyond simple weighted summation, as Multiple objectives in RL adds complexity to the learning process, as policies must balance competing rewards rather than optimizing a single objective. The current implementation should explore methods like Pareto-optimal policy identification by utilizing the insights from "Van Moffaert & Nowé (2014)" to generate a set of non-dominated solutions instead of relying solely on a fixed swing weighting method.

3. Bold Header: Enhanced State Space Modeling

The state space must be expanded to incorporate richer, context-specific information, as the paper notes that a real decision maker would require richer and more context-specific information to act on these insights. Specifically, incorporating energy demand dynamics and regulatory change indicators would allow the agent to move beyond the simplified stochastic models used for prices and production variability.

4. Bold Header: Creative Alternatives Expansion

The action space needs broadening to include non-standard strategic moves, addressing the weakness identified in Figure 11's Decision Quality chain: Creative Alternatives (50%) is arguably the weakest link in this study. This involves adding actions such as withholding investment in unfavorable periods, divesting assets, or allocating funds to other sectors, such as research and development.

5. Bold Header: Uncertainty Modeling Sophistication

The current uncertainty representation should be advanced by replacing simple Gaussian distributions with more context-specific stochastic models derived from historical data for variables like production variability and price dynamics. This aligns with the paper's conclusion that Future research can leverage historical data to build more context-specific stochastic models.