Repurposing Pre-trained LLMs as High Fidelity Continuous Text Autoencoders
cs.LG
Submitted: 2026-09-23
Updated: 2026-09-23
Project page: http://skylion007.github.io/OpenWebTextCorpus
Terminology
Sources
- Understanding Catastrophic Forgetting in Language Models via Implicit Inference
- The Curious Case of Neural Text Degeneration
- Diffusion Models for Non-autoregressive Text Generation: A Survey
- Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution
- Continuous diffusion for categorical data
- Diffusion Transformers with Representation Autoencoders
- Large Language Diffusion Models
- Generalized Interpolating Discrete Diffusion
- Cosmos: Compressed and Smooth Latent Space for Text Diffusion Modeling
- Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models
- Long Context In-Context Compression by Getting to the Gist of Gisting
- In-context Autoencoder for Context Compression in a Large Language Model
- Mistral 7B
- BERTScore: Evaluating Text Generation with BERT
- Vision-Language-Vision Auto-Encoder: Scalable Knowledge Distillation from Diffusion Models
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks