AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling
Jiajun Liang, Yucheng Liao, Yukang Cao, Jiazhe Wei, Ken Li, Wende Tan, Jiankun Zhang, ZY Cui, Jingkang Yang, Liucheng Guo, Shiqi Yang, B. Yang, Caifeng Shan, Ziwei Liu, Chenyang Si
cs.CL
Submitted: 2026-08-03
Comments: 40 pages, 17tables, project page: https://aurora-lm-project.github.io/
Code: https://github.com/fyv587/AURORA-LM
Project page: https://aurora-lm-project.github.io
License: http://creativecommons.org/licenses/by/4.0/
The gist: Language remains an outlier in generative modeling: while images, video, and audio are increasingly modeled in continuous latent spaces, text generation still relies predominantly on discrete tokens.
Terminology
Abstract
Language remains an outlier in generative modeling: while images, video, and audio are increasingly modeled in continuous latent spaces, text generation still relies predominantly on discrete tokens. Existing continuous language models either inherit embedding spaces not designed for joint generation and decoding, or compress autoencoded latents to ease diffusion, sacrificing token-level fidelity. Instead of simplifying the representation to suit the generative model, we preserve a high-capacity, decodable text latent and design the diffusion model to learn its distribution directly. We introduce AURORA-LM, a continuous-latent diffusion language model that separates the construction of a decodable text representation from the modeling of its distribution. A Query-based Encoder-Decoder organizes text into a high-capacity, prefix-aligned latent sequence, and a Block-causal Diffusion Transformer learns its distribution through flow matching, generating blocks left to right while denoising positions within each block in parallel. Because such a latent is harder for diffusion to model, AURORA-LM restricts only the noisy-input pathway while retaining the full clean-latent prediction target, accommodating full-width latents without reducing decoder-facing capacity. We further calibrate the noise-level distribution to the latent width, and introduce self-trajectory consistency to bridge independently sampled training noise and iterative denoising at inference. AURORA-LM achieves the strongest performance among evaluated continuous and diffusion-based language models on OpenWebText free generation and XSum summarization. Scaling to 1B parameters with about 1500 EFLOPs of total compute yields further gains, surpassing a larger publicly released latent-diffusion language model under a matched evaluation protocol. All experiments are conducted on Ascend NPUs.
Sources
- Chameleon: Mixed-Modal Early-Fusion Foundation Models
- LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- DeepSeek-V3 Technical Report
- The Llama 3 Herd of Models
- Continuous Latent Diffusion Language Model
- Classifier-Free Diffusion Guidance
- Video Diffusion Models
- Training Compute-Optimal Large Language Models
- ELF: Embedded Language Flows
- TextLDM: Language Modeling with Continuous Latent Diffusion
- Back to Basics: Let Denoising Generative Models Denoise
- DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models
- Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference
- Large Language Diffusion Models
- GPT-4 Technical Report
- GLU Variants Improve Transformer
- CoDAR: Continuous Diffusion Language Models are More Powerful Than You Think
- Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference
- RoFormer: Enhanced Transformer with Rotary Position Embedding
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering