Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture
cs.LG, cs.CV, stat.ML
Submitted: 2025-06-24
Updated: 2026-09-01
Comments: ICML 2026 Oral Presentation; updated to the camera-ready version
Code: https://github.com/scxue/AO-GPT-MDM
Project page: http://skylion007.github.io/OpenWebTextCorpus
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Score-Based Generative Modeling through Stochastic Differential Equations
- Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution
- Large Language Diffusion Models
- ENTP: Encoder-only Next Token Prediction
- Autoregressive Diffusion Models
- Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling
- Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data
- Scaling Laws for Neural Language Models
- Discrete Copula Diffusion
- Energy-Based Diffusion Language Models for Text Generation
- The LAMBADA dataset: Word prediction requiring a broad discourse context
- One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling
- Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models
- Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions
- Randomized Autoregressive Visual Generation
- RandAR: Decoder-only Autoregressive Visual Generation in Random Orders
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks