Scaling and Distilling Text Embeddings for Better Diffusibility

arXiv:2610.01016 · cs.CL · Submitted 2026-10-01 · Read on arXiv

cs.CL

Submitted: 2026-10-01

Updated: 2026-10-01

Code: https://github.com/Ugness/ELF-pytorch

Terminology

• Embeddings — 116× in this paper · explained in 2 episodes
• Distillation — 26× in this paper · explained in 2 episodes
• Diffusion Model — 22× in this paper · explained in 8 episodes
• Diffusion Models — 16× in this paper · explained in 6 episodes
• Decode — 10× in this paper · explained in 2 episodes
• Latent Space — 10× in this paper · explained in 7 episodes
• Ablation — 5× in this paper · explained in 1 episode
• Foundation Model — 4× in this paper · explained in 3 episodes
• Foundation Models — 4× in this paper · explained in 7 episodes
• Transformer — 4× in this paper · explained in 2 episodes
• Generalization — 3× in this paper · explained in 13 episodes
• Generative AI — 3× in this paper · explained in 2 episodes
• Latent Diffusion Model — 3× in this paper · explained in 2 episodes
• Perplexity — 3× in this paper · explained in 3 episodes
• Representation Learning — 3× in this paper · explained in 2 episodes
• Tokenization — 3× in this paper · explained in 2 episodes
• Accuracy — 2× in this paper · explained in 1 episode
• Adaptation — 2× in this paper · explained in 1 episode
• Benchmark — 2× in this paper · explained in 2 episodes
• Diffusion Process — 2× in this paper · explained in 2 episodes
• Downstream Tasks — 2× in this paper · explained in 2 episodes
• Embedded Language Flow — 2× in this paper
• International Conference on Computer Vision — 2× in this paper
• Latent Diffusion Models — 2× in this paper · explained in 2 episodes
• Self-Distillation — 2× in this paper · explained in 7 episodes
• Adam — 1× in this paper · explained in 2 episodes
• Alignment — 1× in this paper · explained in 1 episode
• Consistency — 1× in this paper · explained in 3 episodes
• Flow Matching — 1× in this paper · explained in 8 episodes
• Generalizability — 1× in this paper · explained in 6 episodes
• Generative Model — 1× in this paper · explained in 2 episodes
• Generative Models — 1× in this paper · explained in 2 episodes
• Hallucination — 1× in this paper · explained in 3 episodes
• Linear probe — 1× in this paper · explained in 2 episodes
• Robustness — 1× in this paper · explained in 5 episodes
• Transfer Learning — 1× in this paper · explained in 10 episodes

Sources

• Why DDIM Hallucinates More Than DDPM: A Theoretical Analysis of Reverse Dynamics
• CoBit: Language Modeling with Bitstream Diffusion
• Seedance 2.0: Advancing Video Generation for World Complexity
• One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling
• Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning
• On the Use of ArXiv as a Dataset
• Continuous diffusion for categorical data
• Continuous Latent Diffusion Language Model
• Diffusion Model's Generalization Can Be Characterized by Inductive Biases toward a Data-Dependent Ridge Manifold
• Distilling the Knowledge in a Neural Network
• Classifier-Free Diffusion Guidance
• ELF: Embedded Language Flows
• TextLDM: Language Modeling with Continuous Latent Diffusion
• DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling
• MCLR: Improving Conditional Modeling via Inter-Class Likelihood-Ratio Maximization and Unifying Classifier-Free Guidance with Alignment Objectives
• Evaluating the Representation Space of Diffusion Models via Self-Supervised Principles
• AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling
• Visual Contrastive Self-Distillation
• Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering