Discrete Beckmann Transport Models for One-Step Language Modeling and Reasoning
cs.LG
Submitted: 2026-09-14
Updated: 2026-09-15
Code: https://github.com/sophtang/DBTM
Project page: http://skylion007.github.io/OpenWebTextCorpus
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
The gist: Discrete diffusion and flow models are a promising alternative to autoregressive language models, but compressing many-step sampling into fewer steps typically requires distilling a pretrained
Terminology
Abstract
Discrete diffusion and flow models are a promising alternative to autoregressive language models, but compressing many-step sampling into fewer steps typically requires distilling a pretrained teacher model. This caps the student at the teacher's quality and requires a costly two-stage training pipeline. We introduce Discrete Beckmann Transport Models (DBTM), built on a time-independent flow whose autonomous transport map provably carries any point in the ambient space to a fixed point on the vertices of the simplex in a single step. We show that this fixed-point property is characterized by a conservation equation whose residual can be minimized directly from data, removing the requirement for a teacher flow and time conditioning. Under this construction, a partially trained map corresponds to the flow truncated at finite time, so generation reduces to iterating one map until it reaches a fixed point. We further extend the map to a partial-context interpolant where additional function evaluations act as refinement steps rather than ODE integration steps. On language modeling and reasoning tasks, DBTM enables one- and few-step generation that improves quality and accuracy over discrete diffusion and continuous flow baselines.
Sources
- Posterior Refinement: Fast Language Generation via Any-Order Flow Maps
- One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling
- LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling
- $\alpha$-Flow: A Unified Framework for Continuous-State Discrete Flow Matching Models
- Training Verifiers to Solve Math Word Problems
- Language Modeling with Hyperspherical Flows
- Continuous diffusion for categorical data
- DeltaFlow: Noise-Adaptive Bidirectional Gated Delta Networks for Embedded Language Flows
- SplitMeanFlow: Interval Splitting Consistency in Few-Step Generative Modeling
- ELF: Embedded Language Flows
- CTRL: A Conditional Transformer Language Model for Controllable Generation
- Flow Map Language Models: One-step Language Modeling via Continuous Denoising
- Beckmann Transport Models: From Autonomous Flows to One-Step Maps
- TinyGSM: achieving >80% on GSM8k with small language models
- Discrete Flow Maps
- Expanding Flow Maps
- Gumbel-Softmax Flow Matching with Straight-Through Guidance for Controllable Biological Sequence Generation
- Continuous Diffusion Scales Competitively with Discrete Diffusion for Language
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks