SPOTting the Future: Lookahead Explanations for Deep Reinforcement Learning

arXiv:2608.09967 · cs.AI, cs.LG · Submitted 2026-07-28 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "SPOTting the Future: Lookahead Explanations for Deep Reinforcement Learning".

Jane: The paper was written by Tamar Gozlan and Claudia Goldman from The Hebrew University of Jerusalem.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title and Authors: Tom: Welcome back to the show, everybody. Today we are cracking open a fresh one from the arXiv: “SPOTting the Future: Lookahead Explanations for Deep Reinforcement Learning.” Jane, I gotta say, that title is doing a lot of work. It’s a pun, it’s an acronym, and it’s a promise.

Jane: It really is, Tom. And the acronym is SPOT, which stands for Sampling Policy Observation Tree. The authors are Tamar Gozlan and Claudia Goldman, both from the Hebrew University of Jerusalem. And right away, the core idea is so refreshing because it’s not trying to explain what the robot is thinking right now. It’s trying to show us where the robot is heading.

Tom: Exactly. And that’s the thing that gets me excited. Most explainability tools for deep reinforcement learning are like looking at a single photograph. They tell you which pixels or which features mattered for one decision. But SPOT is more like a movie trailer. It samples what the agent might do next, simulates those actions, and builds a tree of possible futures.

Jane: Right. And the authors are really clear that this is model-agnostic. So whether you’re using a value-based method like DQN, a policy-based method like REINFORCE, or an actor-critic method like PPO, you can plug SPOT in. They even show how to derive a stochastic policy from a Q-function using a softmax transformation, so the tree construction works everywhere.

Tom: That’s a big deal because a lot of explainability work is tied to one specific algorithm. Here, they’ve built something that just needs a policy you can sample from and a simulator you can step forward. And that’s it. You get this tree that shows the agent’s action preferences and how those preferences play out over a few timesteps.

Jane: And the implications for human oversight are huge. If you have a human operator watching a traffic light controller, they don’t just want to know why the light turned green. They want to know what happens next if it stays green. SPOT gives you that lookahead. It’s a decision-support tool, not just a post-mortem.

Tom: I love that framing. And it sets up the whole paper. They’ve got formal guarantees, they’ve got a real traffic simulation case study, and they’ve got a comparison against SHAP that really exposes the blind spots of single-timestep methods. We’re gonna dig into all of that.

Jane: We absolutely are. And I want to get into the theoretical side next, because they don’t just say “trust me, the tree works.” They actually prove that the tree recovers the most probable action as you sample more.

Tom: Then let’s get there. Stick around, because we’re about to talk about why sampling a hundred actions per node actually gives you a mathematically sound picture of what the agent wants.

Paper Summary: Tom: So we’ve got the title and the authors down. Now let’s talk about what this paper actually does, because the summary is deceptively simple. Jane, you want to break it down?

Jane: Sure. So the whole thing is built around a simple observation: deep reinforcement learning agents are black boxes, but they interact with a simulator. SPOT uses that simulator to ask “what if” questions. At any given state, you sample a bunch of actions from the policy, count how often each action comes up, and then simulate each distinct action to get the next state. Then you repeat that process at each new state, down to a fixed depth.

Tom: And that gives you a tree. The root is the current state, each branch is an action, and each node is a future state. The visit counts on the branches tell you how confident the policy is about each action. High visit count means the policy really wants that action.

Jane: Right. And the beauty is that the tree is interpretable. You can look at it and see not just what the agent chose, but what it almost chose, and what the downstream consequences of each choice look like. The paper calls this a finite-horizon tree, and they cap the depth at K, which in their experiments is three.

Tom: And here’s the part I found really clever. They don’t just stop at visit counts. They store extra information at each node: the critic’s value estimate, the reward, the advantage. So you can see not just which action is likely, but which action is good.

Jane: Exactly. And that’s what makes it useful for explanation. You can compare the chosen path against an alternative path. You can say, “the agent prefers action A, but if it took action B, the expected waiting time would drop.” That’s a contrastive explanation, and it’s something you just can’t get from a feature-attribution method.

Tom: And they prove two theorems about this. The first one says that as you sample more and more actions, the branch with the highest visit count converges to the policy’s true most probable action. So the tree is consistent with the policy.

Jane: And the second theorem is about the flip side. When the policy is nearly uniform, meaning it has no real preference, the probability of the tree picking the wrong action goes to (k-one)/k, where k is the number of actions. So SPOT honestly tells you when the agent is just guessing.

Tom: That’s the kind of honesty you want in an explanation tool. It doesn’t pretend to see structure where there is none. It tells you when the agent is confident and when it’s clueless.

Jane: And that’s a really important property for a human operator. If the tree says “high confidence,” you can trust the recommendation. If it says “this is basically a coin flip,” you know you need to step in.

Tom: So we’ve got the tree, we’ve got the guarantees. Next I want to talk about the actual case study, because they ran this in a traffic simulation and found something really interesting about accidents and sensor blind spots.

Jane: Oh, that’s the best part. They set up a scenario where a car breaks down just outside the sensor range, and the agent can’t see the congestion forming. Let’s get into that.

Improvements Suggested: Tom: Alright, so we’ve covered the core mechanism and the theory. But the paper doesn’t just stop at building the tree. They actually propose a full explanation pipeline on top of it. Jane, what’s the improvement here over just showing someone a tree?

Jane: The improvement is that they turn raw tree data into plain language. They have a four-stage pipeline. First, they decode the raw observation vector into domain concepts. In the traffic case, that means taking the twenty-one-dimensional state and turning it into things like “lane density” and “queue length” for each incoming lane.

Tom: So it’s not just numbers on a screen. It’s actual traffic language.

Jane: Exactly. Then, stage two, they extract two paths from the tree. The chosen path follows the highest-visit branch at every depth, and the counterfactual path starts from the second-highest branch at the first level and then follows the highest-visit branches after that. So you get a “what will likely happen” and a “what would happen if we did something different.”

Tom: And then they look at trends along those paths. They track things like critic values, rewards, and queue lengths over the three steps, and they classify each trend as increasing, decreasing, flat, monotonic, or not.

Jane: Right. And they don’t just report every trend. Stage three is a filtering step. They only keep trends that are strong enough to be meaningful. They compare the magnitude of change against thresholds scaled by the root critic value. If the change is tiny, they drop it. That way the explanation isn’t cluttered with noise.

Tom: And then stage four is the kicker. They compute a stance score. Each piece of evidence votes either for the chosen path or the alternative. Sum it all up, and you get a recommendation: TRUST, WATCH, or INTERVENE.

Jane: And that’s the real improvement over existing methods. SHAP tells you which features mattered. SPOT tells you whether to override the agent. It’s actionable. It’s designed for a human who has the final say.

Tom: And they even include the agent’s confidence in the summary. So if the policy is uncertain, the operator knows. If the alternative action looks better in the lookahead, the operator knows that too.

Jane: And the case study shows exactly when this matters. They created a scenario where a disabled vehicle is upstream of the sensor zone. The agent’s observation looks totally normal, because the queue is outside the sensor range. But SPOT’s lookahead captures the downstream effects.

Tom: Right, because even though the sensor can’t see the queue, the simulated future states reflect the congestion. The tree shows that the alternative phase would reduce expected waiting time, even though the current observation looks fine.

Jane: So the improvement is really about temporal awareness. Single-timestep methods are blind to anything that hasn’t happened yet. SPOT reasons over the next few steps and surfaces problems before they become visible in the raw input.

Tom: And that’s a huge step forward for human-in-the-loop systems. Let’s talk about the actual experiment in detail next, because the numbers they show are pretty compelling.

First Page Discussion: Tom: So we’re deep into the paper now, but let’s step back and look at the very first page, because there’s a lot packed into the abstract and the introduction. Jane, what stood out to you?

Jane: The opening framing is really strong. They say deep reinforcement learning has three big problems: it needs lots of data, it’s sensitive to changing environments, and it lacks interpretability. And they point out that most existing explainability methods are feature-level, meaning they only look at single timesteps.

Tom: And that’s the core critique. Feature attribution tells you which inputs influenced a decision, but it doesn’t tell you how that decision affects the future. The paper calls this out directly: these methods fail to explain long-term consequences or detect suboptimal decisions whose impact emerges over time.

Jane: Right. And that’s exactly the gap SPOT fills. They position it as a human-in-the-loop tool. The agent is a decision-support system, not an autonomous actor. So the explanations have to be actionable. The operator needs to know whether to follow or override the recommendation.

Tom: And they’re very clear about the setting. The agent recommends, the human decides. That changes what an explanation needs to be. It’s not just about understanding. It’s about deciding.

Jane: Exactly. And the abstract mentions formal guarantees. We talked about those already, but it’s worth emphasizing that they don’t just present a heuristic. They prove that the tree recovers the policy’s most probable action in the limit, and they characterize what happens when the policy is uncertain.

Tom: And then they set up the case study. SUMO-RL traffic control. A single intersection, four signal phases, and a reward based on cumulative waiting time. It’s a concrete, real-world domain where decisions have immediate and delayed consequences.

Jane: And the key result is that SPOT uncovers behaviors missed by SHAP. In the accident scenario, SHAP looks at the current observation and says “everything looks normal.” SPOT looks ahead and says “actually, if we keep this phase, waiting times are going to spike.”

Tom: And that’s the moment where you realize the whole point of the paper. It’s not about explaining the past. It’s about anticipating the future. And that’s what makes it a genuinely new contribution.

Jane: I also love that they’re honest about limitations. They note that SPOT doesn’t detect the disruption directly. It captures the downstream effects. The sensor still can’t see the queue. But the lookahead reveals the consequences.

Tom: That honesty is rare. And it makes the tool more trustworthy. You know what it can and can’t do.

Jane: And they close the introduction by saying the framework enables trajectory-aware explanations that explicitly compare outcomes of alternative actions. That’s the whole ballgame.

Tom: So we’ve got the motivation, the method, the guarantees, and the case study. Let’s wrap this up with the big picture and what it means for the field.

Conclusion: Tom: Alright, we’ve spent a lot of time with “SPOTting the Future: Lookahead Explanations for Deep Reinforcement Learning,” and I think we should pull it all together. Jane, what’s the one thing you want listeners to remember?

Jane: The one thing is that SPOT doesn’t just explain a decision. It explains the trajectory. It shows you where the policy is going, not just why it made one move. And that’s a fundamental shift from feature attribution to future awareness.

Tom: And the formal guarantees give you confidence. The tree converges to the true most probable action, and it honestly signals when the policy is uncertain. That’s a tool you can actually rely on in a control room.

Jane: The case study in SUMO-RL really drove it home. A disabled vehicle outside the sensor range, an observation that looks totally normal, and yet SPOT’s lookahead flags the emerging congestion and recommends a different phase. SHAP couldn’t see it because SHAP only looks at the present.

Tom: And the explanation pipeline turns that tree into plain language. TRUST, WATCH, INTERVENE. That’s the kind of clarity a human operator needs when they’re deciding whether to override an agent.

Jane: The authors also mention future work. They want to do user studies to see if these explanations actually change human decisions. That’s the right next step. You can’t just claim actionability. You have to test it with real people.

Tom: And they mention a global behavior summarization component that’s not implemented yet. The idea is to identify critical states across the whole state space and aggregate the SPOT trees rooted there. That could give you a map of the agent’s entire strategy.

Jane: That would be really powerful. Instead of explaining one decision, you’d get a summary of the agent’s strengths and weaknesses across all situations.

Tom: So overall, this paper gives us a new lens for explainability. It’s not about pixels or features. It’s about futures. And that’s a genuinely useful way to think about trusting an agent.

Jane: And with that, we’re going to say goodbye to “SPOTting the Future.” Great paper, great ideas, and we’re excited to see where the follow-up work lands.

Tom: Thanks for listening, everybody. Next up, we’ve got another paper on the stack, and we’ll be back to break it down. Until then, keep asking what happens next.

Tamar Gozlan, Claudia Goldman

The Hebrew University of Jerusalem

cs.AI, cs.LG

Submitted: 2026-07-28

Updated: 2026-08-12

Code: https://github.com/LucasAlegre/sumo-rl

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 48/100

The gist: for policy-based and actor-critic methods, the policy is directly available; for value-based methods, a stochastic policy is derived via a softmax transformation of the Q-function: π(ais) = exp(Q(s,

Key concepts

SPOT (Sampling Policy Observation Tree)
This method builds a tree by sampling actions from the agent's policy and simulating the resulting next states. It tracks how often each action is chosen and then repeats this process down to a fixed depth, showing all potential paths and their consequences stemming from the agent's current decision.
Lookahead Explanation
Instead of analyzing why an agent made a single choice (feature attribution), this approach simulates what happens next. It shows the trajectory—the sequence of future states and outcomes—allowing users to understand long-term effects that might not be visible in the initial observation.
Explainability Pipeline
This is a four-stage process that converts raw data into plain language. It identifies paths (chosen vs. alternative), tracks trends (increasing, decreasing) over time, filters out noise, and finally computes a 'stance score' to recommend whether the human should TRUST, WATCH, or INTERVENE.

Terminology

Summary

Summary

The paper introduces SPOT (Sampling Policy Observation Tree), a novel, model-agnostic, sampling-based framework for interpreting deep reinforcement learning (DRL) policies. SPOT addresses a key limitation of existing explainable AI (XAI) methods for DRL, which largely focus on feature-level attribution and are inherently limited to single timesteps and do not capture how decisions affect future states. The authors state that such methods fail to explain long-term consequences or detect suboptimal decisions whose impact emerges over time. SPOT overcomes this by modeling the distribution of future trajectories induced by the agent’s policy and constructing an interpretable tree through action sampling and state expansion, enabling trajectory-aware explanations that explicitly compare the outcomes of alternative actions.

The framework is designed for a human-in-the-loop setting where the agent acts as a decision-support system, requiring explanations that are actionable, enabling operators to assess whether to follow or override recommendations. SPOT provides domain-readable summaries that highlight key signals to support such decisions.

Methodology and Theoretical Guarantees. SPOT assumes access to a stochastic policy π(as) over a finite action space and an environment simulator E. It is model-agnostic: for policy-based and actor-critic methods, the policy is directly available; for value-based methods, a stochastic policy is derived via a softmax transformation of the Q-function: π(ais) = exp(Q(s, ai)) / Σj exp(Q(s, aj)). The tree construction (Algorithm 1) proceeds as follows: starting from a root node representing the current state s, SPOT draws N independent action samples from π(·s). Instead of creating a branch per sample, it aggregates identical actions, computing visit counts Ci = Σn 1 An = ai. For each action with Ci > 0, the simulator generates the successor state s′i = E(s, ai), and a child node is created with the branch labeled (ai, Ci). This process is applied recursively to each node until a maximum depth K is reached, producing a finite-horizon tree of the policy’s local decision behavior.

The paper provides two formal theorems. Theorem 1 (Consistency with the Greedy Policy) states that if a⋆ = arg maxa π(as) is the unique greedy action, then Pr(arg maxai Ci = a⋆) → 1 as N → ∞, establishing that SPOT recovers the greedy action in the limit of large sample sizes. Theorem 2 (Disagreement Under Maximum Uncertainty) states that as the policy approaches the uniform distribution π(ais) → 1/k, the disagreement probability satisfies Pr(TN ≠ a⋆) → (k−1)/k, characterizing the behavior of SPOT under high-entropy policies, where action preferences become indistinguishable.

Extension for Rich Policy Analysis. SPOT's tree structure can store auxiliary quantities beyond visit counts, including the critic’s value estimate V(s), action-value estimates Q(s,a), advantage estimates A(s,a), and observed rewards along sampled trajectories. These are computed during tree expansion. The framework supports several explanation methods from prior work [Goldman et al., 2025]: Plan Next explanations describe the agent’s expected future behavior by identifying high-probability trajectories in the tree; Contrastive explanations compare alternative branches to explain why a selected action is preferred over other feasible actions; and Value Component explanations decompose value-related quantities stored in the tree. The paper also notes a potential extension for Global Behavior Summarization by identifying critical states where different actions lead to substantially different outcomes, though this is not implemented in the current work.

Case Study: Traffic Signal Control. SPOT is demonstrated in a SUMO-RL traffic control environment with a single four-way signalized intersection. The agent's task is to dynamically select the traffic signal phase at each timestep so as to minimize cumulative vehicle waiting time. The observation space is a 21-dimensional vector (phase one-hot, min-green indicator, per-lane densities and queues). The action space consists of four discrete signal phases. A PPO agent is trained for 100,000 timesteps using Stable-Baselines3. SPOT is configured with N=100 samples per node and depth K=3.

Explanation Extraction Pipeline. The pipeline converts raw tree data into human-readable text in four stages:

  1. Observation decoding: The 21-dimensional observation is decoded into domain concepts over eight incoming lanes.

  2. Path extraction and trend detection: Two paths are extracted—the chosen path follows the highest-visit child at every depth (Plan-Next explanation), and the counterfactual path starts from the depth-1 node with the second-highest visit count (Contrastive explanation). For each path, six scalar sequences are extracted (critic values, rewards, advantages, policy confidence, queue lengths, densities). Each sequence is classified into one of seven trend categories (e.g., monotone-large-positive, non-monotone-net-negative, flat) based on net change relative to thresholds scaled by the root critic value and trajectory shape.

  3. Signal-strength filtering: Trends that lack sufficient magnitude or clear directional structure are suppressed.

  4. Stance scoring and actionable summary: The chosen path is compared against the alternative path via signed votes, producing a scalar score mapped to a recommendation: T RUST (high positive), WATCH (intermediate), or I NTERVENE (strongly negative). The summary includes the stance, key signals, agent confidence, and, for I NTERVENE, the preferred alternative phase.

Accident Scenario Demonstrating Future-Aware Reasoning. The authors design a scenario to expose the blind spot of attribution-only methods like SHAP. At t=600, a stationary vehicle is placed on the northbound lane upstream of the sensor coverage region, causing a queue to form outside the observed region. Because SUMO-RL measures traffic only near the stop line, the corresponding queue feature remains light throughout the accident window, even as congestion builds upstream. SPOT does not detect the disruption directly but captures its downstream effects through lookahead. At step 122, despite the agent strongly favoring the current phase, SPOT recommends I NTERVENE, identifying an alternative phase that reduces expected future waiting time, even though the observed state appears normal. At step 172, SPOT recommends T RUST, confirming the agent's action leads to favorable downstream outcomes. The paper contrasts this with SHAP, which cannot account for how traffic conditions evolve over time or how alternative actions would affect future states and fails to detect the emerging congestion.

Conclusion. The paper concludes that SPOT faithfully reflects the underlying policy (with theoretical guarantees), is model-agnostic, and uncovers robustness issues arising from partially observable disruptions by reasoning over future trajectories, enabling more informed intervention in a human-in-the-loop setting. Future work includes user studies to evaluate the actionability of SPOT explanations.

Improvements for AI systems

Based on the SPOT framework described in the paper, here are the specific improvements I can implement in an AI system, along with what the improved system can do:

Implementation: Replace single-timestep feature attribution (e.g., SHAP) with a recursive sampling tree that expands N=100 stochastic policy samples to depth K=3 at each decision point. Each node stores visit counts, rewards, critic values, and advantages.

Resulting capability: The AI system can now explain why an action was chosen by showing the downstream consequences of that action versus alternatives, rather than merely listing which input features were influential. For example, in traffic control, it can state: Choosing phase 2 leads to a 40% increase in queue length on the southbound lane within 15 seconds, whereas phase 3 reduces it by 25%.

Implementation: Extract two paths from each tree: (a) the highest-visit path (chosen trajectory) and (b) the second-highest-visit depth-1 branch followed by highest-visit children (alternative trajectory). Compute per-step deltas in rewards, critic values, and advantages between these paths.

Implementation: For each extracted path, compute six scalar sequences (critic values, rewards, advantages, policy confidence, queue lengths, densities). Classify each sequence into one of seven trend categories (monotone-large-positive, monotone-small-positive, monotone-large-negative, monotone-small-negative, non-monotone-net-positive, non-monotone-net-negative, flat) using thresholds scaled to 20% and 60% of the root critic value.

Implementation: Apply a filter that suppresses weak or ambiguous trends (small fluctuations, inconsistent directional patterns) and retains only trends with sufficient magnitude or clear directional structure, as defined by the thresholds in Table 2 of the paper.

Implementation: Aggregate signed votes from all filtered trends into a single scalar score. Map the score to one of three stances: T RUST (high positive), WATCH (intermediate), or I NTERVENE (strongly negative). When I NTERVENE, identify the preferred alternative phase and explain why it is favored.

Implementation: Although SPOT uses the same sensor-based observations as the agent (and thus cannot directly see upstream blockages), it captures the downstream effects of such disruptions through the tree's lookahead. By comparing critic values and rewards across branches, the system detects when the current action leads to deteriorating future conditions even when the current observation appears normal.

Implementation: For value-based methods (e.g., DQN), derive a stochastic policy via softmax over Q-values: π(as) = exp(Q(s,a)) / Σ exp(Q(s,a')). For policy-based and actor-critic methods, use the actor's output directly.

  1. Provide forward-looking explanations: Instead of explaining only what features drove this decision, it explains what will happen if we follow this decision versus alternatives over a 15-second (3-step) horizon.

  2. Support operator override decisions: It outputs a clear stance (TRUST/WATCH/INTERVENE) with quantitative justification, enabling rapid, informed human intervention in safety-critical domains.

  3. Detect latent failures: It identifies situations where the current action leads to future degradation even when the current observation appears normal—a capability absent from single-timestep attribution methods.

  4. Generate natural-language summaries: It converts complex tree data into concise, domain-readable paragraphs that highlight key signals, trends, and recommendations, reducing the operator's cognitive burden.

  5. Compare alternative futures explicitly: It quantifies the difference between the chosen trajectory and the most plausible alternative, allowing operators to see exactly what they gain or lose by overriding the agent.

  6. Operate across DRL architectures: It works with value-based, policy-based, and actor-critic methods without requiring architectural changes, making it a drop-in explanation module for existing systems.

  7. Filter noise and focus on actionable signals: It suppresses weak trends and only reports changes that exceed meaningful thresholds, preventing information overload and false alarms.

These improvements transform the AI system from a black box that recommends actions into a decision-support system that explains consequences, compares alternatives, and flags risks—directly enabling the human-in-the-loop oversight required for real-world deployment in domains like traffic control, healthcare, and autonomous systems.

Sources

Related papers