Math for AI safety: an invitation for mathematicians

arXiv:2609.15289 · math.HO, cs.AI, cs.LG · Submitted 2026-09-14 · Read on arXiv

math.HO, cs.AI, cs.LG

Submitted: 2026-09-14

Updated: 2026-09-14

Comments: 33 pages, 5 figures

Code: https://github.com/ColombanD/open-source-game-theory

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Cooperation — 13× in this paper · explained in 2 episodes
• Generalization — 13× in this paper · explained in 13 episodes
• Transformer — 12× in this paper · explained in 2 episodes
• Interpretability — 9× in this paper · explained in 15 episodes
• Reinforcement Learning — 8× in this paper · explained in 4 episodes
• AI Agents — 7× in this paper · explained in 2 episodes
• Steering — 7× in this paper · explained in 2 episodes
• Active Inference — 3× in this paper · explained in 1 episode
• Alignment — 3× in this paper · explained in 1 episode
• Benchmark — 3× in this paper · explained in 2 episodes
• Core — 3× in this paper · explained in 2 episodes
• Gradient Flow — 3× in this paper · explained in 2 episodes
• Regret — 3× in this paper · explained in 3 episodes
• Trace — 3× in this paper · explained in 2 episodes
• World Model — 3× in this paper · explained in 7 episodes
• AI Alignment — 2× in this paper · explained in 2 episodes
• Activation Steering — 2× in this paper · explained in 4 episodes
• Axioms — 2× in this paper · explained in 2 episodes
• Conference on Learning Theory — 2× in this paper
• Mechanistic interpretability — 2× in this paper · explained in 2 episodes
• Sparsity — 2× in this paper · explained in 4 episodes
• Action Space — 1× in this paper · explained in 1 episode
• Activation Patching — 1× in this paper · explained in 2 episodes
• Adam — 1× in this paper · explained in 2 episodes
• Adaptation — 1× in this paper · explained in 1 episode
• Consistency — 1× in this paper · explained in 3 episodes
• Decoupling — 1× in this paper · explained in 2 episodes
• Deep Reinforcement Learning — 1× in this paper · explained in 2 episodes
• Degeneracy — 1× in this paper · explained in 3 episodes
• Distribution Shift — 1× in this paper · explained in 6 episodes
• Fisher Information — 1× in this paper · explained in 3 episodes
• Generative Model — 1× in this paper · explained in 2 episodes
• Multi-Agent System — 1× in this paper · explained in 6 episodes
• Phase Transitions — 1× in this paper · explained in 2 episodes
• Predictive Coding — 1× in this paper · explained in 2 episodes
• Reconstruction Error — 1× in this paper · explained in 2 episodes
• Reward Function — 1× in this paper · explained in 2 episodes
• Variational Bayes — 1× in this paper · explained in 2 episodes
• Verifiable Rewards — 1× in this paper · explained in 2 episodes

Sources

• SGD learning on neural networks: leap complexity and saddle-to-saddle dynamics
• Remarks on the disproof of the unit distance conjecture
• Concentration of tempered posteriors and of their variational approximations
• Refusal in Language Models Is Mediated by a Single Direction
• Occam's razor is insufficient to infer the preferences of irrational agents
• Robust Cooperation in the Prisoner's Dilemma: Program Equilibrium via Provability Logic
• Sparse Representation of a Polytope and Recovery of Sparse Signals and Low-rank Matrices
• A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations
• Characterising Simulation-Based Program Equilibria
• Parametric Bounded L\"ob's Theorem and Robust Cooperation of Bounded Agents
• Cooperative and uncooperative institution designs: Surprises and problems in open-source game theory
• Sparse Autoencoders Find Highly Interpretable Features in Language Models
• Toy Models of Superposition
• On the uniqueness and stability of dictionaries for sparse representation of noisy signals
• Logical Induction
• Sparse and spurious: dictionary learning with noise and outliers
• Cooperative Inverse Reinforcement Learning
• Progress in Formalizing Sphere Packing in Dimension 8
• When can dictionary learning uniquely recover sparse data from subsamples?
• The Bayesian Learning Rule

Related papers

• A unified interpretation of probability
• Remembering Solomon Marcus
• LLAMA LIMA: A Living Meta-Analysis on the Effects of Generative AI on Learning Mathematics
• A machine-checked proof of the Dong-Yang classification of optimal (n,4) binary codes for BSCs
• If you can distinguish, you can express: Galois theory, Stone--Weierstrass, machine learning, and linguistics
• Explanations, Prompts, and Formalizations: Arguments for New Norms in LLM-Enabled Mathematical Research