2608.09335-Control-Oriented Scenario Tree Construction through Reinforcement Learning

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Control-Oriented Scenario Tree Construction through Reinforcement Learning".

Jane: The paper was written by Fabio Pavirani, Bert Claessens, Pierre Pinson and Chris Develder from Ghent University and Beebop.ai and Imperial College London.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary: Tom: We just introduced this paper in the cold open, and honestly I'm still buzzing. The whole premise is that when you build a scenario tree for stochastic control, you've been optimizing the wrong thing. A controller facing an uncertain future approximates that future with a branching tree of possible paths, and the standard recipe builds that tree by matching the probability distribution as closely as possible.

Jane: And the paper's argument is that a tree which matches the distribution well isn't automatically a tree that makes good decisions. Two price trajectories can look close together in ordinary Euclidean distance and still call for completely opposite control actions. So distributional fidelity is a poor proxy for whether the tree actually helps the controller.

Lu: Exactly. So instead of minimizing a statistical distance, they train a policy to build the tree directly on the realized control profit. The tree's shape is fixed in advance, but the policy decides which sampled scenarios get assigned to which leaves. Then they optimize that assignment with reinforcement learning, where the reward is the actual closed-loop profit of a battery arbitrage controller.

Meng: What I love is that this isn't model-free RL replacing the optimizer. The exact, constraint-respecting, risk-averse optimization program stays in the loop the whole time. The learning is only shaping the uncertainty input that the program reasons over. That's a really clean division of labor.

Lalam: And the results justify the effort. On a battery arbitrage task, the learned constructor beats classical forward and backward scenario reduction, and beats certainty-equivalent control, at every fan size up to two hundred sampled trajectories. The trees it builds are also compact — it populates about two of its six available leaves on average. Better profit and a smaller optimization problem at the same time.

Tom: The most striking part for me is the tail. Its worst-case profits stay positive across the whole sweep of fan sizes, while certainty-equivalent control goes negative at several of them. That's exactly what a risk-averse operator would ask for.

Jane: So the headline is: the value of a scenario tree depends on the decisions it supports, and you can learn that value directly instead of approximating it with a probability metric. That's the thesis. I want to go back to page one now, where they set up the problem from the ground up.

Page 1 of the paper: Tom: The introduction reads like a critique of a default habit in stochastic optimization. The habit is: build the scenario tree by minimizing a probability distance to the forecast, and trust that a statistically faithful tree will give you good decisions. The paper says that assumption has no basis in the control problem.

Jane: Right, and the context is concrete. Modern power systems are full of renewables, volatile prices, and flexible demand, so planning against a single forecast is fragile. The standard response is multistage stochastic MPC, where you optimize over many possible futures at once. But that requires compressing the continuous forecast distribution into a finite scenario tree, and that's where the trouble starts.

Lu: Because the tree determines both the size of the optimization problem and how uncertainty is presented to the controller. If you construct it by pure distribution matching, you never ask whether the scenarios you keep actually matter for the decisions at hand. Those classical methods come with stability guarantees, but the guarantees bound how far the optimal value can drift — they don't say whether a different tree would have given a better decision.

Meng: Wait, so the distance-based reduction is trying to solve a different problem than the controller actually cares about?

Lu: That's exactly it. The reduction is agnostic to the downstream optimization. And that's the real intellectual shift in this work — they don't add a clever correction to the distribution-matching recipe. They replace the objective altogether, training the tree constructor on the closed-loop control profit and nothing else.

Jane: The abstract states it plainly. The learned policy exhibits greater robustness and better tail-risk characteristics, and the trees carry compact, selectively branching structures that keep most trajectories nearly deterministic. It's a bold set of promises.

Lalam: What impresses me is the framing. The author list spans Ghent University, a company called Beebop.ai, and Imperial College London, and you can feel both the power-systems side and the machine-learning side in the writing. They take a combinatorial problem — assigning scenarios to branches — and turn it into a sequential decision process that reinforcement learning can handle. That reframing is the real contribution.

Tom: And they promise to beat classical reduction, keep better tail risk, and build compact trees that keep the optimization cheap. That's a strong menu. The natural question is how they position themselves against everyone who has tried something similar — and that's exactly where the paper goes next.

Page 2 of the paper: Tom: Before the paper introduces its own method, it spends real effort carving out a position among previous attempts. There's a whole line of problem-driven scenario reduction — Bertsimas and Mundru, Hewitt and colleagues, Zhuang and colleagues in power systems — that tailors the reduction to the optimization problem rather than to the distribution. But those are one-shot or iterative procedures, usually for two-stage formulations, and they must be re-run every time the forecast changes.

Jane: That's the key contrast. The policy here is an amortized constructor — trained once, then reused at every control step without solving an auxiliary reduction problem. The training signal is the realized closed-loop profit from actually rolling out the controller. No surrogate objective, no per-instance optimization. And they're careful to distinguish themselves from decision-focused learning too: the learning target is the discrete tree-building operation, where gradients don't flow, not the forecast itself.

Lu: The positioning against model-free reinforcement learning is sharp as well. Earlier work replaces the battery dispatcher entirely with a learned policy that has to discover constraints like state-of-charge limits from reward signals alone. That gives you no feasibility guarantees. Here it's the opposite — the exact risk-averse optimizer stays in charge, and learning only shapes the uncertainty representation it consumes.

Meng: Then the problem formulation section moves into the concrete testbed. A grid-connected battery doing arbitrage, with a fixed planning horizon, charge and discharge efficiencies, power and capacity limits. Given a known price trajectory, the dispatch problem is just a linear program. In closed loop, you solve it over the look-ahead window, apply the first action, advance one step, and resolve.

Lalam: And that receding-horizon loop is what makes the training signal honest. The paper doesn't evaluate a tree in isolation; it measures how the tree performs when an actual controller lives with it, day after day. Later experiments put the perfect-foresight oracle at roughly thirty-seven hundred in profit — an unattainable upper bound — which gives every other method a clear reference point.

Jane: So the deterministic dispatch is the easy case. The hard part is the stochastic setting, where all you have is a fan of sampled trajectories and you have to compress them into a branching tree. That formulation, and then the sequential assignment method the authors build on top of it, is what we're headed into now.

Page 3 of the paper: Tom: The construction method hinges on a simple trick. You fix the tree's shape in advance — the root, the leaves, the branches connecting them — and the only thing left to decide is which sampled scenario goes to which leaf. Because each leaf defines a root-to-leaf path, placing a scenario in a leaf determines every node it travels through.

Jane: And that single assignment operation encodes non-anticipativity automatically. Scenarios sent to the same leaf are treated identically up to the branching point; scenarios sent to different leaves diverge at exactly the stage their paths separate. So decisions can only depend on information available at that point, which is precisely the constraint a multistage controller needs. The construction respects it by construction.

Lu: The paper is also careful about pruning. A leaf that receives no scenario is removed, and any branch left empty disappears with it. So the policy isn't just deciding how scenarios cluster; it's implicitly deciding how many branches the final tree truly uses. In the extreme, routing everything to one leaf collapses the controller back to certainty-equivalent dispatch.

Meng: That's a nice observation — the fixed topology still allows the effective tree to vary.

Lu: Exactly. And the sequential element is clever too. Assigning all scenarios at once would be a huge combinatorial action, so the paper groups them. Scenarios are sorted by how far they deviate from the mean trajectory, the unusual ones first, then split into groups. At each step the policy sees the whole fan plus the partial assignments and only places the current group.

Tom: The bookkeeping lives in the scenario tokens: which leaf each scenario is assigned to so far, with an extra entry for unassigned, and a flag marking which scenarios are being decided right now. Those two fields carry the state of the partially built tree from one group step to the next.

Jane: And at the end of the sequence, the completed tree feeds the optimizer, the first action is applied, and the environment returns the arbitrage profit as the reward. So the reward is literally the closed-loop control performance, and that's what the policy optimizes. Which raises the obvious question — what kind of network can actually read an unordered set of scenarios, remember what's been assigned, and output good leaf choices? That's the architecture discussion.

Page 4 of the paper: Tom: The architecture is where the machine-learning pedigree shows. The policy has to be permutation-equivariant — the fan is a set, so the order of scenarios shouldn't matter. Every scenario becomes a token that packs the battery state, the whole price trajectory, its probability, and the two bookkeeping fields we just talked about: the assigned leaf so far, and whether it's in the group being placed right now.

Jane: Then there's a smart efficiency trick. Instead of letting every scenario attend to every other scenario, only the current group's tokens act as queries. They attend over the full context of the entire fan. That brings the per-step cost down to order G times S instead of S squared, which matters a lot at the larger fan sizes they test.

Lu: The actor outputs a softmax over the leaves for each scenario in the group, and the group distribution factorizes over its members. During training they sample assignments; at deployment they take the argmax. The critic mirrors the same encoder but reads out a single scalar value from a global token — a value estimate for the partial construction state.

Meng: And the critic has one extra privilege. During training its tokens include the realized future trajectories, which the actor never sees. That's the asymmetric critic. It gives a lower-variance value target, and since the critic is discarded at deployment, the deployed controller still relies on nothing beyond the forecast.

Tom: The training procedure is very much standard PPO: horizon-length undiscounted returns, a clipped surrogate with an entropy bonus, a small replay buffer to stay roughly on-policy, and early stopping when the policy drifts too far from the behavior policy. They even normalize the per-scenario log-probabilities by the group size so the group actions are comparable.

Jane: What I appreciate is the engineering honesty. They want the learning signal to come from the actual optimization solver, so a single training rollout means solving a multistage program at every time step. That's expensive — which is exactly why they train at a small fan size of ten and then test whether the policy generalizes to much larger ones. Which brings us to the experiments, and the synthetic price environment they built to make those experiments meaningful.

Page 5 of the paper: Tom: The experiments run in a synthetic electricity price environment, and the paper is upfront about why. They're not trying to reproduce a specific market; they want a controlled setting where uncertainty genuinely matters. The price process combines three ingredients: mean reversion toward a deterministic twenty-four-hour cycle, continuous noise, and rare jump events that create heavy tails.

Lu: That jump component is crucial, because the whole motivation is tail risk. The latent state feeds through a nonlinear transform, so a large shock becomes an amplified price spike. If a scenario tree can't represent those rare extreme trajectories, a risk-averse controller has no way to hedge against them.

Meng: The setup is clean. Two hundred training profiles and two hundred held-out evaluation profiles, each a hundred and twenty steps long. At every control step the controller gets a fan of S scenarios over a six-step horizon, drawn from the same underlying process with uniform probabilities. So the forecast is correct by construction — any performance difference comes from how the tree is built, not from forecast bias.

Jane: And the baselines are chosen to isolate exactly what the learned policy contributes. Oracle is perfect foresight, the unattainable upper bound. Deterministic is the element-wise mean, certainty-equivalent control. Forward and backward are the classical distance-based reductions at a leaf budget of six. Random uses the same six-leaf topology but assigns scenarios uniformly, so it isolates the value of learned assignment from the value of the topology itself.

Tom: The main metric is realized cumulative profit, with tail metrics like CVaR at five and ten percent, plus the size of the resulting tree and the wall-clock time to build and solve it. The policy is trained once at S equals ten and then evaluated unchanged all the way up to S equals three hundred. That's a real generalization test.

Lu: Right — if the policy only memorized the small-fan setting, the larger fans would expose it immediately. So the numbers in the results section are the payoff of all that design work. Let's look at them.

Page 6 of the paper: Tom: The headline numbers are in the profit table, and they hold up at every fan size up to two hundred. At S equals ten, the RL agent earns fourteen thirty-four, versus thirteen fifteen for backward reduction and twelve thirty-five for deterministic. At two hundred, it's sixteen forty-five, with backward at fifteen ninety-seven and deterministic at sixteen oh-seven. Only at three hundred does backward marginally edge ahead, sixteen forty-four to sixteen thirty-eight — and that's deep inside overlapping error bars.

Jane: But the more interesting picture is the gap-closed figure, where zero is certainty-equivalent control and a hundred is the oracle. The learned policy is the only method that stays non-negative across the entire sweep. Backward reduction dips below the deterministic reference at the intermediate fan sizes, and forward selection sits well below it from S equals fifty onward, down around minus eight to minus twelve percent.

Lu: That forward result makes sense given the algorithm. Forward selection greedily picks scenarios that are far from the ones already chosen, which favors spread over representativeness. As the fan grows, its six retained leaves drift toward atypical trajectories. The learned policy doesn't have that failure mode because it adapts how many leaves it actually populates.

Meng: The per-profile comparison is where I got convinced. They plot the agent's profit against each baseline, one point per profile, and the win rate — the fraction of profiles where the agent earns more — is above fifty percent in every single panel. It peaks at seventy-nine percent against deterministic at the smallest fan, and it stays above fifty even at three hundred, where the mean profit is essentially tied.

Jane: There's also that curious random baseline. Uniform assignment performs far better than you'd expect, hovering around the deterministic reference and actually beating it on the lower tail at every fan size. The paper reads that as implicit hedging — six randomly drawn scenarios are an unbiased sample of the forecast, so the controller is forced to hedge across realistic variability.

Tom: So the means converge at large fans, but convergence hides a lot. The next part digs into the tails and the actual trees the policy builds — and that's where the risk-averse story gets its strongest evidence.

Page 7 of the paper: Tom: The structural results explain everything. Even though the topology offers six leaves, the learned policy populates only about two of them on average after pruning. Backward and forward always use all six. So the RL agent is essentially certainty-equivalent control with a small, selective amount of branching — which is exactly why it tracks the deterministic baseline from above.

Lu: And that compactness pays off in solver time. The learned tree's linear program solves in roughly three and a half to four and a half milliseconds per step, about half the cost of the six-leaf trees. But the build time is the bigger story. Backward reduction's pairwise distance computations grow fast, overtaking the agent around S equals one hundred fifty and reaching about four point eight seconds at three hundred.

Meng: Meanwhile, the RL build time

Conclusion: Tom: So, to wrap it all up: this paper takes scenario-tree construction for stochastic MPC and turns it from a distribution-matching problem into a reinforcement-learning problem, trained purely on the closed-loop control profit.

Jane: And the results really do speak for themselves. The learned policy beats classical forward and backward reduction, and beats certainty-equivalent control, everywhere that matters in practice — especially in the small-sample regime where online optimization stays tractable.

Tom: What convinced me was the tail behavior. The agent's worst-case profit stays positive across the entire sweep of fan sizes, while the deterministic controller dips to or below zero multiple times. Same expected profit or better, with substantially less downside. That's what a risk-averse operator actually wants.

Jane: And the structural story makes it intuitive. The policy uses only about two of its six leaves on average — it's essentially certainty-equivalent control with a small, selective amount of branching. It hedges only when hedging helps, which is why it tracks the deterministic baseline from above instead of falling below it.

Tom: The authors are also honest about the limits. The dispatch problem is linear, so certainty-equivalent control approaches optimality as the fan grows. The forecast is correctly specified, which is the friendliest setting for both distance-based reduction and deterministic control. The strongest case will come from misspecified forecasts and nonlinear problems.

Jane: And that's exactly where I'd expect the learned approach to shine even more. If the forecast is biased, a method trained on control profit rather than forecast fidelity has room to compensate. On nonlinear problems, Jensen's inequality means a well-constructed tree keeps its value at every fan size. The case is promising, but the harder settings will be the real proof.

Tom: We should also remember the engineering angle — the build cost grows more slowly than backward reduction's, so in the convergence region you reach the same performance with fewer scenarios and cheaper construction. That kind of practical advantage matters as much as the profit numbers.

Jane: We'll be right back after a short break with a paper that tackles stochastic control under exactly the kind of nonlinear dynamics these authors flagged as their next frontier. Stay with us.