Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey

arXiv:2507.22920 · cs.CL, cs.AI · Submitted 2025-07-21 · Read on arXiv

cs.CL, cs.AI

Submitted: 2025-07-21

Updated: 2026-09-21

Comments: Published in IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)

DOI: 10.1109/TPAMI.2026.3676982

Code: https://github.com/jindongli-Ai/LLM-Discrete-Tokenization-Survey

Project page: https://byteflow-ai.github.io/TokenFlow

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

Quantization — 222× in this paper · explained in 12 episodes
Tokenization — 117× in this paper · explained in 2 episodes
Vector Quantization — 53× in this paper
Transformer — 44× in this paper · explained in 2 episodes
Alignment — 35× in this paper · explained in 1 episode
Product Quantization — 27× in this paper
Representation Learning — 16× in this paper · explained in 2 episodes
Embeddings — 13× in this paper · explained in 2 episodes
Residual Vector Quantization — 12× in this paper
Fidelity — 11× in this paper · explained in 2 episodes
Knowledge Graph — 11× in this paper · explained in 7 episodes
Regularization — 11× in this paper · explained in 3 episodes
Foundation Model — 10× in this paper · explained in 3 episodes
Generative Recommendation — 9× in this paper · explained in 2 episodes
Binary Spherical Quantization — 6× in this paper
Clustering — 6× in this paper · explained in 2 episodes
Diffusion Model — 6× in this paper · explained in 8 episodes
Finite Scalar Quantization — 6× in this paper
Optimized Product Quantization — 6× in this paper
Robustness — 6× in this paper · explained in 5 episodes
Vector Quantization (VQ) — 6× in this paper · explained in 2 episodes
Additive Vector Quantization — 5× in this paper
Composite Quantization — 5× in this paper
Foundation Models — 5× in this paper · explained in 7 episodes
Knowledge graphs — 5× in this paper · explained in 2 episodes
Contrastive Learning — 4× in this paper · explained in 5 episodes
Downstream Tasks — 4× in this paper · explained in 2 episodes
Fine-Tuning — 4× in this paper · explained in 5 episodes
Graph Anchor-Relation Tokenization — 4× in this paper
Interpretability — 4× in this paper · explained in 15 episodes
Large Language Models (LLMs) — 4× in this paper · explained in 6 episodes
Latent Representation — 4× in this paper · explained in 2 episodes
Transferability — 4× in this paper · explained in 4 episodes
Abstraction — 3× in this paper · explained in 1 episode
Differentiable Product Quantization — 3× in this paper
Diffusion Models — 3× in this paper · explained in 6 episodes
Generalization — 3× in this paper · explained in 13 episodes
Gradient Flow — 3× in this paper · explained in 2 episodes
Latent Representations — 3× in this paper · explained in 3 episodes
Latent Space — 3× in this paper · explained in 7 episodes
Look-up Free Quantization — 3× in this paper
Sparsity — 3× in this paper · explained in 4 episodes
Straight-Through Estimator — 3× in this paper
Supervised Quantization — 3× in this paper
Transformed Residual Quantization — 3× in this paper
Accuracy — 2× in this paper · explained in 1 episode
Adaptation — 2× in this paper · explained in 1 episode
Core — 2× in this paper · explained in 2 episodes
Distillation — 2× in this paper · explained in 2 episodes
GNNs — 2× in this paper · explained in 2 episodes
Generative Modeling — 2× in this paper · explained in 3 episodes
Grounding — 2× in this paper · explained in 3 episodes
In-Context Learning — 2× in this paper · explained in 4 episodes
Locally Optimized Product Quantization — 2× in this paper
Multimodal Learning — 2× in this paper · explained in 2 episodes
Product Tree Quantization — 2× in this paper
Residual Vector Product Quantization — 2× in this paper
Semantic Alignment — 2× in this paper · explained in 3 episodes
Variational Bayes — 2× in this paper · explained in 2 episodes
World Model — 2× in this paper · explained in 7 episodes
Adversarial Training — 1× in this paper · explained in 1 episode
Attention Mechanism — 1× in this paper · explained in 2 episodes
Chain-of-Thought Prompting — 1× in this paper · explained in 2 episodes
Computational overhead — 1× in this paper · explained in 2 episodes
Consistency — 1× in this paper · explained in 3 episodes
Decode — 1× in this paper · explained in 2 episodes
Decoupling — 1× in this paper · explained in 2 episodes
Exponential Moving Average — 1× in this paper
Exponential Moving Average (EMA) — 1× in this paper · explained in 2 episodes
Federated Learning — 1× in this paper · explained in 14 episodes
Generative Model — 1× in this paper · explained in 2 episodes
Generative Models — 1× in this paper · explained in 2 episodes
Hybrid Architectures — 1× in this paper · explained in 3 episodes
Instruction Tuning — 1× in this paper · explained in 2 episodes
Knowledge Distillation — 1× in this paper · explained in 11 episodes
LLM Integration — 1× in this paper · explained in 2 episodes
Modularity — 1× in this paper · explained in 9 episodes
Multi-modal Inputs — 1× in this paper · explained in 2 episodes
Multimodal Fusion — 1× in this paper · explained in 3 episodes
Multimodal Integration — 1× in this paper · explained in 2 episodes
Persistent Homology — 1× in this paper · explained in 3 episodes
Personalization — 1× in this paper · explained in 2 episodes
Redundancy — 1× in this paper · explained in 2 episodes
Regret — 1× in this paper · explained in 3 episodes
Reinforcement Learning — 1× in this paper · explained in 4 episodes
Self-Supervised Learning — 1× in this paper · explained in 7 episodes
Semantic Understanding — 1× in this paper · explained in 2 episodes
Transformer Models — 1× in this paper · explained in 2 episodes
Unified Framework — 1× in this paper · explained in 5 episodes
Variational Autoencoder (VAE) — 1× in this paper · explained in 3 episodes
alpha — 1× in this paper · explained in 1 episode

Sources

Categorical Reparameterization with Gumbel-Softmax

Terminology

Sources

Related papers