Application of machine learning to monster level prediction in tabletop RPG game design

arXiv:2607.09196 · cs.LG · Submitted 2026-07-10 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Application of machine learning to monster level prediction in tabletop RPG game design".

Jane: The paper was written by Jolanta Śliwaa and Jakub Adamczyka from AGH University of Krakow, Faculty of Computer Science, Cracow, Poland.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Summary: Tom: So, the authors used this massive dataset of six thousand seven monster entries from Pathfinder Second Edition to test their approach. It’s a huge collection!

Jane: They framed the whole task as "tabular ordinal regression," which is just a fancy way of saying that predicting the monster's level isn't just about guessing a number, but respecting the order of the classes.

Lu: I think understanding that distinction is crucial; it acknowledges that being Level four should be more than just being two steps higher than Level two.

Meng: And they managed to build this first dedicated dataset for TTRPG prediction, which is a huge practical contribution, since data availability is often the biggest hurdle in ML research.

Lalam: The sheer scale of the data they compiled suggests that this project has the potential to scale up to help with any rule-based system, not just Pathfinder.

Improvements and Methodology: Tom: One of the most important parts of their methodology was how they engineered those features; it's not just taking raw stats.

Jane: They looked at things like armor class and hit points, but also factored in things like maximum spell level or attack bonuses to create more meaningful vectors for the extended feature set.

Lu: This shows a sophisticated understanding that the game mechanics are interconnected; you can’t look at one variable in isolation when trying to determine overall power.

Meng: The way they handled features is very practical, though; it's a lot of work turning those raw JSON blocks into these clean tabular inputs for machine learning models.

Lalam: It suggests that we are moving toward a deeper level of automated design where the AI understands the underlying logic of the world-building itself, rather than just making random guesses.

Results and Implications: Tom: The results were pretty striking; they found that tree-based ensembles like Random Forest performed exceptionally well compared to linear models or neural networks.

Jane: It’s amazing to hear that, because it means the "non-linear" complexity of game design is something these algorithms handle naturally.

Lu: I think this success points toward a future where designers can finally rely on a data-driven tool that understands the subtle interactions between systems.

Meng: From an engineering standpoint, the fact that tree models excelled suggests that for tabular data like this, robustness and ensemble methods are more reliable than complex deep learning architectures.

Lalam: The implications here is huge; we might be entering an era where AI helps us create balanced worlds without sacrificing the human creativity of having a powerful monster.

Conclusion: Tom: It sounds like the overall conclusion is that machine learning can reliably approximate designer judgments for these specific game systems.

Jane: And it's not just about prediction, but providing a practical, tool-like assistance for monster balancing.

Lu: I’m excited to see the future work mentioned, especially using counterfactual generation to suggest what designers want to change in stats.

Meng: That counterfactual idea is extremely practical; telling a designer exactly what changes are needed is far more useful than just giving them a prediction score.

Lalam: The entire effort of "Application of machine learning to monster level prediction in tabletop RPG game design" gives us hope that AI can help structure our imaginative worlds in a way that is both creative and consistent.

Jolanta Śliwaa, Jakub Adamczyka

AGH University of Krakow · Faculty of Computer Science, AGH University of Krakow, Cracow, Poland

cs.LG

Submitted: 2026-07-10

Updated: 2026-08-24

Code: https://github.com/tunczyk101/Monster-level-prediction-in-TTRPG

Importance score: 91/100

The gist: This paper investigates whether machine learning can support tabletop role-playing game (TTRPG) designers by predicting a monster's ordinal level from its numerical attributes.

Key concepts

Tabular Ordinal Regression
This is a statistical method used to predict a monster's level while respecting the natural order of the classes. It acknowledges that advancing from one level to the next represents a significant increase in power and complexity, rather than just being a simple numerical jump.
Feature Engineering
This methodology involves creating meaningful inputs for machine learning models that go beyond raw statistics. The authors factored in elements like maximum spell levels or attack bonuses to create more comprehensive vectors for determining overall game power.
Tree-based Ensembles
These are types of machine learning algorithms, such as Random Forest, that combine multiple decision trees to achieve predictions. The episode notes they performed exceptionally well on the tabular data, suggesting robustness and reliability for this type of analysis.

Terminology

Summary

This paper investigates whether machine learning can support tabletop role-playing game (TTRPG) designers by predicting a monster's ordinal level from its numerical attributes. Because designing balanced adversaries is a central but labor-intensive task, automating this process provides an effective computer-aided tool for monster balancing and broader TTRPG system design.

The dataset and feature engineering

To address the need for automation, the authors created what is, to our knowledge, the first dataset built specifically for TTRPG monster-level prediction, utilizing 6007 entries from Pathfinder Second Edition. The task is formalized as a tabular ordinal regression problem because a monster's level is both discrete and ordered but does not exist on a ratio scale. To facilitate learning, the researchers developed three distinct feature sets:

  • Basic: Consists of 8 fundamental statistics including basic attributes (Str, Dex, Con, Int, Wis, Cha), Hit Points (HP), and Armor Class (AC).

  • Extended: Adds 12 characteristics related to defensive abilities, offensive capabilities, and magical power.

  • Full: Includes 33 features by adding complex characteristics like movement speeds, immunities, and granular spell-count data.

Model selection and evaluation protocols

The study compares 16 different models to determine the most effective approach for predicting power levels. These include:

  1. Classical regression models using various rounding schemes.

  2. Dedicated tabular ordinal regression algorithms, such as Ordered Random Forest (ORF) and Gaussian Process Ordinal Regression (GPOR).

  3. Neural networks utilizing specialized architectures and ordinal-aware losses, such as CORN and CONDOR.

To ensure a realistic quantification of model generalization, the authors avoided standard random splits, which can cause data leakage due to the temporal nature of game design. Instead, they employed two domain-specific protocols: a chronological split based on publication dates and an expanding-window evaluation strategy that mimics how designers incorporate new books into their workflow.

Experimental results and conclusions

The experimental results demonstrate that tree-based ensembles outperform linear models and neural approaches. Specifically, models like Random Forest and LightGBM achieved near-perfect ordinal ranking and high predictive accuracy. While neural networks were capable of outperforming human-inspired baselines, they generally showed higher variance and were less effective for this specific tabular dataset compared to tree-based methods.

Explainable AI (XAI) analyses, including feature importance via SHAP values, revealed that the models are aligned with human intuition. The most influential predictors were defensive statistics—such as Armor Class (AC), Hit Points (HP), and saving throws—and offensive melee capabilities. Ultimately, the authors conclude that machine learning can reliably approximate designer judgments and serve as a robust tool for TTRPG system design.

Improvements for AI systems

1. Tree-Ensemble Based Procedural Balancing Engine

  • Improvement: Replace standard Multi-Layer Perceptrons (MLPs) or LLM-based statistical estimation with a hybrid architecture utilizing Tree-Based Ordinal Regression Ensembles (specifically Random Forest and LightGBM) as the core mathematical validator.

  • Capability: This system can act as an automated Mathematical Auditor for procedurally generated content. It will verify that any generated NPC or monster adheres to the specific power-scaling laws of a rule system, ensuring that difficulty levels are assigned with high rank-consistency (Somers' D) and minimal error in imbalanced distributions (low Macro-MAE), preventing stat bloat or underpowered encounters in automated game design.

2. SHAP-Driven Counterfactual Design Assistant

  • Improvement: Integrate SHAP (SHapley Additive exPlanations) feature importance analysis with a counterfactual optimization loop applied to the trained Random Forest model.

  • Capability: This system allows designers to perform Targeted Attribute Tuning. A designer can input a desired level (e.g., Level 12) and a base concept (e.g., High AC, Low HP), and the AI will suggest the minimal, mathematically optimal adjustments to specific attributes (like increasing Spell DC or adjusting Melee Attack Bonus) required to hit that exact target level, drastically reducing manual iteration time.

3. Temporal-Aware Generative Content Pipeline

  • Improvement: Implement an Expanding-Window Training Protocol for generative models used in game development, rather than standard random data splitting.

  • Capability: This system can simulate the real-world evolution of a game's power curve. By training on chronologically ordered datasets, the AI learns to mimic the design drift and mathematical progression patterns found in professional rulebook releases, allowing it to generate new content that feels like a natural, balanced successor to existing material rather than an outlier.

4. Ordinal-Fitness Evolutionary PCG (Procedural Content Generation)

  • Improvement: Replace standard Mean Squared Error (MSE) fitness functions in evolutionary algorithms with Macro-Averaged MAE and Somers' D rank correlation metrics.

  • Capability: This enables the evolution of highly specialized Boss monsters. Most AI systems fail at the tails of a distribution (the very high or very low levels); by using these specific ordinal metrics, the AI can evolve complex, high-level adversaries that are mathematically precise and statistically significant, ensuring that rare and powerful creatures are as balanced as common ones.

Sources

Related papers