Attention-Weighted Value Projection for KV-Cache Compression
cs.LG, cs.AI, cs.CL
Submitted: 2026-04-13
Updated: 2026-09-08
Comments: 10 pages, 3 figures. Corrected general optimality and cache-storage claims; exact fixed-attention value-projection theorem and equal-cost allocation corollary. Detailed changes in Appendix A. No new model evaluations
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM
- KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelity
- AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
- KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
- Pointer Sentinel Mixture Models
- MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projection
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks