When Fancy Eviction Fails: Rethinking Cache Replacement For LLM Prefix Reuse

arXiv:2609.28870 · cs.DC, cs.LG · Submitted 2026-09-24 · Read on arXiv

cs.DC, cs.LG

Submitted: 2026-09-24

Updated: 2026-09-24

Code: https://github.com/cacheMon/cache_

Terminology

• Trace — 98× in this paper · explained in 2 episodes
• KV cache — 12× in this paper · explained in 3 episodes
• Core — 8× in this paper · explained in 2 episodes
• Transformer — 7× in this paper · explained in 2 episodes
• Decode — 2× in this paper · explained in 2 episodes
• Quantization — 2× in this paper · explained in 12 episodes
• Symposium on Operating Systems Design and Implementation — 2× in this paper
• USENIX Annual Technical Conference — 2× in this paper
• Agentic Coding — 1× in this paper · explained in 1 episode
• Alignment — 1× in this paper · explained in 1 episode
• Attention Mechanism — 1× in this paper · explained in 2 episodes
• Benchmark — 1× in this paper · explained in 2 episodes
• Benchmarking — 1× in this paper · explained in 3 episodes
• GPU High-Bandwidth Memory — 1× in this paper
• IEEE International Symposium on Workload Characterization — 1× in this paper
• Instruction Tuning — 1× in this paper · explained in 2 episodes
• Integer Linear Programming — 1× in this paper
• Least Recently Used — 1× in this paper
• Mixture-of-Experts (MoE) — 1× in this paper · explained in 7 episodes
• Mixture-of-Experts (MoE) Models — 1× in this paper · explained in 3 episodes
• Multi-Layer Perceptron — 1× in this paper
• Robustness — 1× in this paper · explained in 5 episodes
• Scaling Laws — 1× in this paper · explained in 4 episodes
• Sparsity — 1× in this paper · explained in 4 episodes

Sources

• State of AI: An Empirical 100 Trillion Token Study with OpenRouter
• Scaling Laws for Neural Language Models
• Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models
• Strata: Hierarchical Context Caching for Long Context Language Model Serving

Related papers

• iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
• SAMM: Sharded Automated Market Maker
• InferScale: GPU-Native KV Injection for Personalized LLM Serving
• Vigil: Accountable Liveness against Selective Silence
• Steelhead: Interleaving Partially Synchronous and Asynchronous Commit Rules on a Shared DAG
• Pushing CPU Speech Synthesis to the Wall: Extreme Inference Tuning under Serverless Architecture and Billing