CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration
cs.LG, cs.AI
Submitted: 2026-08-31
Updated: 2026-08-31
Comments: Published at ICML 2025
Journal ref: Proceedings of the 42nd International Conference on Machine Learning (ICML), PMLR 267:27569-27585, 2025
Code: https://github.com/haoyun-jiang/CateKV
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large language models (LLMs) have demonstrated strong capabilities in handling long-context tasks, but processing such long contexts remains challenging due to the substantial memory requirements and
Terminology
Abstract
Large language models (LLMs) have demonstrated strong capabilities in handling long-context tasks, but processing such long contexts remains challenging due to the substantial memory requirements and inference latency. In this work, we discover that certain attention heads exhibit sequential consistency in their attention patterns, which can be persistently identified using a coefficient-of-variation-based algorithm. Inspired by this observation, we propose CateKV, a hybrid KV cache method that retains only critical token information for consistent heads, thereby reducing KV cache size and computational overhead, while preserving the majority of KV pairs in adaptive heads to ensure high accuracy. We show the unique characteristics of our algorithm and its extension with existing acceleration methods. Comprehensive evaluations on long-context benchmarks show that, while maintaining accuracy comparable to full attention, CateKV reduces memory usage by up to 2.72 times and accelerates decoding by 2.18 times in single-sample inputs, and boosts throughput by 3.96 times in batch scenarios.
Sources
- Peek Across: Improving Multi-Document Modeling via Cross-Document Question-Answering
- PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling
- MagicPIG: LSH Sampling for Efficient LLM Generation
- The Llama 3 Herd of Models
- Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference
- LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference
- Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
- Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning
- GPT-4 Technical Report
- Qwen Technical Report
- ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools
- LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
- Loki: Low-rank Keys for Efficient Sparse Attention
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference
- MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention
- RazorAttention: Efficient KV Cache Compression Through Retrieval Heads
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
- Gemini: A Family of Highly Capable Multimodal Models
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks