Non-Stationarity Breaks Permutation Surrogates in Multi-Agent Reinforcement Learning: Diagnosis and Remedies

arXiv:2604.23716 · cs.AI, cs.IT, cs.LG, cs.MA, math.IT · Submitted 2026-04-26 · Read on arXiv

cs.AI, cs.IT, cs.LG, cs.MA, math.IT

Submitted: 2026-04-26

Updated: 2026-09-08

Comments: 28 pages, 4 figures. Substantially revised: rebuilt around a controlled experiment with ground-truth null and positive controls. Code and data at https://doi.org/10.5281/zenodo.22658530 and https://github.com/dentros/te-nonstationarity

Code: https://github.com/dentros/te-nonstationarity

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Mutual Information — 20× in this paper
• Transfer Entropy — 14× in this paper
• Failure Modes — 11× in this paper · explained in 3 episodes
• Reinforcement Learning — 10× in this paper · explained in 4 episodes
• Representation Learning — 10× in this paper · explained in 2 episodes
• Q-learning — 9× in this paper · explained in 3 episodes
• Animat — 6× in this paper · explained in 1 episode
• Feature Selection — 5× in this paper · explained in 4 episodes
• Active Inference — 4× in this paper · explained in 1 episode
• Non-stationarity — 4× in this paper · explained in 2 episodes
• Regularization — 4× in this paper · explained in 3 episodes
• Core — 3× in this paper · explained in 2 episodes
• Curse of Dimensionality — 3× in this paper · explained in 6 episodes
• International Conference on Machine Learning — 3× in this paper
• Actor — 2× in this paper · explained in 3 episodes
• Benchmark — 2× in this paper · explained in 2 episodes
• Calibration — 2× in this paper · explained in 7 episodes
• Generative Modeling — 2× in this paper · explained in 3 episodes
• Interpretability — 2× in this paper · explained in 15 episodes
• Multi-Agent System — 2× in this paper · explained in 6 episodes
• Predictive Coding — 2× in this paper · explained in 2 episodes
• Robustness — 2× in this paper · explained in 5 episodes
• Accuracy — 1× in this paper · explained in 1 episode
• Agent Architectures — 1× in this paper · explained in 1 episode
• Bayesian Inference — 1× in this paper · explained in 5 episodes
• Computational Biology — 1× in this paper · explained in 2 episodes
• Contrastive Learning — 1× in this paper · explained in 5 episodes
• Deep Reinforcement Learning — 1× in this paper · explained in 2 episodes
• Distillation — 1× in this paper · explained in 2 episodes
• Epistemic Uncertainty — 1× in this paper · explained in 9 episodes
• Generative AI — 1× in this paper · explained in 2 episodes
• Generative Model — 1× in this paper · explained in 2 episodes
• Generative Models — 1× in this paper · explained in 2 episodes
• Hilbert Space — 1× in this paper · explained in 2 episodes
• IEEE Information Theory Workshop — 1× in this paper
• Information Gain — 1× in this paper · explained in 2 episodes
• KL Regularization — 1× in this paper · explained in 2 episodes
• Knowledge Distillation — 1× in this paper · explained in 11 episodes
• Kullback-Leibler (KL) Divergence — 1× in this paper · explained in 3 episodes
• Latent Space — 1× in this paper · explained in 7 episodes
• Loss Function — 1× in this paper · explained in 2 episodes
• Modular Architecture — 1× in this paper · explained in 4 episodes
• Redundancy — 1× in this paper · explained in 2 episodes
• Selection Bias — 1× in this paper · explained in 4 episodes
• Self-Supervised Learning — 1× in this paper · explained in 8 episodes
• Variational Bayes — 1× in this paper · explained in 2 episodes
• Wasserstein Distance — 1× in this paper · explained in 4 episodes

Sources

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Learning deep representations by mutual information estimation and maximization
• Auto-Encoding Variational Bayes
• Estimation of Shannon differential entropy: An extensive comparative review
• Trust Region Policy Optimization

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
• VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection