More GPUs or a Smaller Cache? Tensor Parallelism versus KV Compression for Memory-Bound LLM Serving
cs.AI
Submitted: 2026-08-25
Updated: 2026-08-25
Terminology
Sources
- MLPerf Inference Benchmark
- Vidur: A Large-Scale Simulation Framework For LLM Inference
- Efficient Memory Management for Large Language Model Serving with PagedAttention
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Ring Attention with Blockwise Transformers for Near-Infinite Context
- Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
- DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
- KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
- KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
- H$_2$O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
- EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Serving
- KV Cache Transform Coding for Compact Storage in LLM Inference
- Rethinking Key-Value Cache Compression Techniques for Large Language Model Serving
- LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scale
- AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
- PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling
- Fast Inference from Transformers via Speculative Decoding
- Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection