Tensor Decomposition of Transformer Key-Value Caches: Spectral Structure and Format Comparison

arXiv:2609.28029 · math.NA, cs.CL, cs.LG, cs.NA · Submitted 2026-09-23 · Read on arXiv

math.NA, cs.CL, cs.LG, cs.NA

Submitted: 2026-09-23

Updated: 2026-09-23

Code: https://github.com/rahulk98/JoLT-Master-Thesis

Terminology

• Allocator — 14× in this paper · explained in 1 episode
• KV cache — 14× in this paper · explained in 3 episodes
• Reconstruction Error — 9× in this paper · explained in 2 episodes
• Transformer — 9× in this paper · explained in 2 episodes
• Ablation — 5× in this paper · explained in 1 episode
• Core — 3× in this paper · explained in 2 episodes
• Aggregation — 2× in this paper · explained in 1 episode
• Generative AI — 2× in this paper · explained in 2 episodes
• In International Conference on Learning Representations — 2× in this paper
• Tucker Decomposition — 2× in this paper · explained in 2 episodes
• Empirical Methods in Natural Language Processing — 1× in this paper
• Generalization — 1× in this paper · explained in 13 episodes
• Transformer Models — 1× in this paper · explained in 2 episodes
• Verification — 1× in this paper · explained in 2 episodes

Sources

• Fast Transformer Decoding: One Write-Head is All You Need
• Mistral 7B
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation

Related papers

• Do physics-informed neural networks (PINNs) need to be deep? Shallow PINNs using the Levenberg-Marquardt algorithm
• A Neural-preconditioned Poisson Solver for Mixed Dirichlet and Neumann Boundary Conditions
• Second-order consistency for learning chaotic dynamics via randomized Jacobian matching
• Windowed thinning and query complexity for the bouncy particle and Zigzag samplers
• Data-efficient Kernel Methods for Learning Hamiltonian Systems
• Adjoint Method versus Physics-Informed Neural Networks in PDE-Constrained Inverse Problems