2608.07220-Beyond the Black Box: Interpretable Models of Human Randomisation Failures

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Beyond the Black Box: Interpretable Models of Human Randomisation Failures".

Jane: The paper was written by Ngoc Linh Dao from Alpen-Adria-University of Klagenfurt.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary: Tom: We've got a fascinating one today — a paper that takes a classic economics experiment and asks whether transparent models can match what a black-box neural network can predict. It's all about why humans are so bad at being unpredictable.

Jane: And the short answer is yes, mostly, with a really interesting caveat about what that predictability actually is. The authors use data from a card game where people should play randomly, and they recover most of the predictive power of an LSTM using simple, readable models.

Lu: The dataset is huge by experimental economics standards — 84,060 decisions from 2,802 pairs over 30 rounds. And the game is O'Neill's four-card game, which has a sharp theoretical benchmark: in equilibrium, choices should be independent across rounds.

Meng: So any detectable pattern is, by definition, a departure from the benchmark. The question is what kind of departure — do people avoid repeating themselves, do they track their opponent, are they learning payoffs, or are they chasing frequencies?

Lalam: And the paper's answer is that one mechanism dominates: people manage their own recent action histories — repeat and avoid behaviour. That accounts for most of the explainable signal, and it's also the strategically exploitable signal.

Tom: So it's not deep strategic sophistication hiding in the data. It's a simpler vulnerability — players trying to cover their tracks, and in doing so leaving traces of exactly what they did.

Jane: The headline numbers are striking. Their extended EWA model recovers about 89 percent of the LSTM's improvement over a constant baseline for Red players, and 98 percent for Black players. That's with a model you can open up and read.

Lu: And it matters beyond this one game, because being unpredictable shows up in poker, in security, in any setting where a human has to randomise against an adversary. If the failure is structured, that structure is exploitable.

Meng: I also appreciate that they're honest about the limits — they admit the improved models might have captured different regularities than the LSTM and just happened to perform similarly.

Lalam: Which is exactly the right caution. The paper moves the debate from "can we predict" to "what exactly are we predicting" — and that's a much more useful question.

Tom: To see how they got there, we should go back to the beginning of the paper, where the game and the puzzle are laid out in detail.

Page 1: Tom: So page one sets up the game carefully. There are two roles — Red and Black — and four cards: the numbers one, two, three, and the face card K.

Jane: Red wins if both players pick K, or if both pick different number cards. Black wins if exactly one player picks K, or if both pick the same number card. It's zero-sum, so one player's win is the other's loss.

Lu: And the equilibrium is sharp: each number card should be played with probability 0 point 2, and K with probability 0 point 4, independently every round. That independence is the key benchmark — if past actions help predict future choices, you've left the equilibrium.

Meng: The paper points to an old tension in the literature. O'Neill's original 1987 study found aggregate play was close to the minimax benchmark, but Brown and Rosenthal in 1990 re-examined and showed individual choices were still history-dependent.

Lalam: And that's the gap this paper sits in — the aggregate looks fine, but the individual sequences leak information. Recently, Hirasawa and colleagues used LSTMs to predict those deviations out of sample, and they did very well.

Tom: Which raises the core question, stated right there on page one: what is that predictability made of? Is it own-action habits, opponent tracking, payoff learning, or frequency tracking?

Jane: They also borrow a line from Cynthia Rudin — the way forward is to design models that are inherently interpretable, rather than explaining black boxes after the fact. That's the methodological stance of the whole paper.

Lu: I find it telling that all models are estimated separately by role, because the equilibrium itself is asymmetric — Red wins 40 percent of the time, Black 60 percent. So the authors don't assume the two roles fail in the same way.

Meng: That also means the incentive structure differs by role. If you're Black and winning more often, the pressure to randomise correctly might feel different than if you're Red.

Lalam: Right, and the page closes with three promises: replicate the interpretable-versus-black-box comparison, enrich the feature analysis, and extend the behavioural model with an ML-guided term. That last bit is where the paper gets original.

Jane: So the stage is set. The next page is where they lay out the machinery — the different model families and how they're all forced to do the same prediction task.

Page 2: Tom: Page two gets technical, but the structure is clean. Every model has to solve the same task: given the history up to round t minus one, predict the card at round t, as probabilities over the four cards.

Jane: And they all produce those probabilities the same way — through a multinomial logit, so the probability of a card is proportional to the exponential of some scoring function. The models differ only in how they summarise the history.

Lu: The simplest baseline is the empirical constant model — the i.i.d. model. It just uses the observed card frequencies and ignores history entirely. That's the benchmark every other model has to beat.

Meng: Then they add serial correlation — the probability of a card shifts if the player chose that same card in one of the previous n rounds. They test orders one and four, and order four predicts best, along with a restricted version where number cards share coefficients.

Lalam: And then comes Experience-Weighted Attraction — EWA — the classic Camerer and Ho model from 1999. Each card has an attraction that updates after every round, blending what you actually got with what you would have got had you played differently.

Tom: The update rule has parameters controlling how much past experience is discounted, how much past attractions decay, and how weight falls on foregone payoffs. Then a sensitivity parameter maps the attractions to choice probabilities.

Jane: What I like is that EWA nests both reinforcement learning and belief learning as special cases. So it's a flexible umbrella, and yet the paper will show it recovers very little of the LSTM's predictive power.

Lu: That's a genuinely important result on its own — the standard learning model, which economists have relied on for decades, mostly misses whatever is driving these sequence patterns. The structure is somewhere else.

Meng: And the estimation is maximum likelihood with constraints to keep the parameters in sensible ranges — rho and delta between zero and one, lambda non-negative. All fairly standard econometrics so far.

Lalam: Right, and that contrast sharpens on the next page, when the machine learning models show up — LASSO, decision trees, DNNs, LSTMs — followed by the modified EWA family that tries to bridge the two worlds.

Tom: Exactly. Let's look at page three and see how that bridge is built.

Page 3: Tom: Page three introduces four machine learning models. Two are interpretable — a LASSO multinomial logit and a decision tree — and two are neural benchmarks, a DNN and an LSTM.

Jane: The LASSO and the tree share the same enriched feature set: lagged actions, streaks, joint action profiles, win-loss histories, recent card counts, best-response indicators. That's the raw material for the interpretable analysis.

Lu: The DNN and LSTM replicate what Hirasawa and colleagues did — the DNN looks at a fixed window of the last four actions, while the LSTM reads the whole action sequence recurrently. They're deliberately high-capacity, not meant to explain anything, just to measure how much structure is extractable.

Meng: And then there's the modified EWA family, which is where the paper's own contribution comes in. ME1 adds own repeat-and-avoid terms — whether the player just played a card, or seems to be avoiding it.

Lalam: ME2 adds an opponent term, tracking whether the opponent repeatedly played or avoided the cards that the focal card beats. That's the payoff-relevant opponent history, and it follows Hirasawa's specification.

Tom: And ME3 is the new piece — it translates the frequency-tracking features the LASSO selected into a behavioural term. It essentially checks whether the player best-responds to the opponent's modal card over the last three, four, or five rounds.

Jane: So the logic is: let the machine learning point you at candidate mechanisms, then build a small, readable model that embodies that mechanism, and test whether it helps out of sample. That's the ML-guided approach in action.

Lu: One detail I found interesting — the starred variants extend the memory length T, and out of sample the performance forms a flat plateau, with optima around ten rounds for Red and fourteen for Black. So memory length is only weakly identified — the model doesn't care much exactly how far back you look.

Meng: And they mention the fitting uses a JAX-based optimiser with multiple random starts, because some of these high-dimensional variants are otherwise prohibitively hard to fit. That's a practical point worth remembering.

Lalam: So the full toolkit is assembled — classics, behavioural models, machine learning, and this hybrid family. The next page is where everything gets compared head to head.

Tom: Let's get to the numbers, then. Page four has the results table.

Page 4: Tom: Page four opens with the evaluation metrics, and they're worth spelling out because they measure different things. KL divergence is just the negative log probability assigned to the realised action, so it matches the objective the models were trained on.

Jane: The strategic error rate is more interesting — it's the win rate of the focal player if the opponent best-responded to the model's prediction. Lower means the model's prediction is more exploitable strategically.

Lu: And relative completeness is the headline metric — the fraction of the LSTM's improvement over the constant baseline that a given model recovers. Zero is the baseline, one is the LSTM.

Meng: So what do the results show? First, Nash — the equilibrium model — actually does worse than the empirical constant model. Playing the theoretical mix is worse than just using the observed frequencies.

Lalam: And standard EWA recovers very little of the LSTM gap, which confirms what we suspected from page two — the classic learning model is the wrong tool here. Serial correlation at order four does better, but still captures only a limited share.

Tom: Then the machine learning models: LASSO is the surprise star — it reaches relative completeness of 0 point 769 for Red and 0 point 869 for Black. A sparse, readable set of history variables recovers most of the black-box signal.

Jane: The decision tree, by contrast, performs only modestly. That's a nice negative result — a small set of if-then rules isn't enough to capture the behaviour. The structure is richer than a handful of rules.

Lu: And the LSTM remains the strongest model overall, beating everything by a wide margin. So there is still something the interpretable models don't fully reach — but the gap is much smaller than you'd expect.

Meng: Then the modified EWA results: ME1 jumps to 0 point 769 for Red and 0 point 802 for Black — the own repeat-and-avoid terms are doing the heavy lifting. ME2 star closes most of the remaining gap, reaching 0 point 888 and 0 point 980.

Lalam: Though we should note the paper's caution there — ME2 star also gets longer memory lengths, so part of the gain could come from memory rather than the opponent-tracking mechanism itself. And ME3 star barely improves on it: 0 point 895 for Red, unchanged for Black.

Tom: So the ranking is clear — own history dominates, opponent payoff-relevant history adds something, frequency tracking adds almost nothing. And the strategic error rates fall steadily along the same order.

Jane: Which sets up page five, because there they go beyond the model comparisons and look directly at which features the LASSO chose, to check the behavioural ranking from a completely different angle.

Page 5: Tom: Page five has that model-agnostic check I just mentioned — Table 2, where the LASSO-selected features are grouped into families with behavioural interpretations. And the ranking from the model comparisons holds up.

Jane: The own repeat-and-avoid features are selected most consistently — variables like "the player did not play card a" come in positive, and just-played indicators come in negative. Long streaks for number cards and for K enter with different signs.

Lu: That difference between number cards and K matters, because K has a different equilibrium probability — 0 point 4 versus 0 point 2. Players seem to treat the face card as a different kind of object, and the model supports estimating them separately.

Meng: The secondary channel is opponent history — variables for whether the opponent repeatedly played or avoided the cards that the focal card beats. That matches the ME2 improvement, and it's payoff-relevant, not just generic opponent tracking.

Lalam: And the weak channel is frequency tracking — best-response-to-opponent-mode indicators do get selected, but with small coefficients. So the LASSO independently points to the same conclusion: modal tracking exists, but it's not the main driver.

Tom: I also notice outcome dependence being listed — recent wins and losses get selected too. But the paper is careful to say that doesn't imply loss overweighting; outcomes matter, but the evidence doesn't support a specific behavioural bias like that.

Jane: And because the same pattern appears for both Red and Black players, the ranking is unlikely to be an artifact of one role. That's a nice robustness check — the structure is symmetric even though the equilibrium isn't.

Lu: Then there's the strategic error rate story. SER falls steadily from the constant model through ME1 to ME2 star and ME3 star, approaching the LSTM. So the predictive gains are strategically meaningful — a best-responding opponent would actually make fewer errors against these predictions.

Meng: That connects the prediction task back to the real-world worry — if you can predict a human's randomisation, you can exploit it. The interpretable models aren't just academically interesting; they'd be practically dangerous as opponents.

Lalam: And that's why the conclusion on page six has to grapple with what all this says about the black-box question and about human behaviour more generally.

Tom: Let's close it out, then — the conclusion is where they confess the limits as well as celebrate the wins.

Conclusion: Tom: So the conclusion — the answer to the question we started with is largely yes. The interpretable models do recover most of the black-box predictive power: about 89 percent of the LSTM's KL improvement for Red players, and 98 percent for Black players.

Jane: But they're refreshingly honest about a lingering doubt — whether the improved models actually decoded what was inside the black box, or captured different regularities that happen to perform similarly. That's a genuinely open question.

Lu: On the behaviour side, the main signal is repeat-and-avoid. Players manage their own recent action sequences, and that's where the largest chunk of predictability comes from. EWA on its own explains very little.

Meng: The opponent's payoff-relevant history adds a bit more, and modal frequency tracking adds almost nothing out of sample. So ME3's value is mostly diagnostic — it rules out a plausible mechanism and thereby strengthens the repeat-and-avoid interpretation.

Lalam: And the LASSO independently corroborates that ranking, which is the methodologically satisfying part — two very different tools point at the same behavioural structure.

Tom: The broader implication is a bit humbling for human strategic skill. We think of unpredictability as a strategic ability, but the paper suggests the failures are structured and exploitable in a simple way — people leave traces of their own actions while trying to hide them.

Jane: For applications — poker, security, any adversarial setting — that means the exploitable signal might not require a massive neural network to find. A readable model can get you most of the way there.

Lu: And that dovetails with the interpretability agenda — Rudin's argument that for high-stakes decisions you want models you can inspect, not black boxes you can only trust.

Meng: I'd love to see this tested in other games, with other equilibrium structures, to see whether the repeat-and-avoid dominance is specific to this card game or something more general about how humans randomise.

Lalam: That's the natural next step. For now, the paper gives us a clear answer to its own question, an honest caveat about what remains unknown, and a neat template for how machine learning can guide behavioural theory instead of just outperforming it.

Tom: And with that, we'll say goodbye to this one — plenty to chew on. Next up, we've got another paper waiting, so let's take a short break and come back fresh.

Jane: Sounds good — see you in a moment.