InferScale: GPU-Native KV Injection for Personalized LLM Serving

summary

The gist

In short

InferScale is a GPU-native system that injects personalized LLM memory directly into vLLM's cache to eliminate repeated prompt prefilling. It achieves near-constant time-to-first-token regardless of context size by precomputing and reusing KV states for retrieved memory facts, maintaining high retrieval accuracy.

Key concepts

Context-Window Encoding: This technique encodes each stored memory fact along with a small window of previous conversation turns. This ensures that when a fact is retrieved, it retains enough local context to be accurate without needing to recompute the entire prompt or fine-tune the model.
Chunked RoPE: This method stores keys before rotary position encoding and then re-rotates them at insertion time during serving. This allows a precomputed fact's KV representation to be placed at any position in the sequence while ensuring it produces identical attention scores as if it had been prefilled there originally.
KV Injection via vLLM KV Connector: This is the mechanism that places the precomputed memory directly into vLLM's paged cache. It bypasses traditional prefill steps by registering the memory during scheduling and performing a direct GPU-to-GPU copy, ensuring tokens are ready immediately.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Federated Learning — explained in 16 episodes
Retrieval-Augmented Generation (RAG) — explained in 16 episodes
Interpretability — explained in 15 episodes
Sample Complexity — explained in 15 episodes
Generalization — explained in 13 episodes

This episode discusses

The paper

InferScale: GPU-Native KV Injection for Personalized LLM Serving · Read on arXiv

Northeastern University

Large language models are increasingly deployed with persistent personalized context, such as accumulated memory profiles or long conversation histories, that is shared across a user's many requests. Production memory systems (e.g., Mem0, MemGPT, and Zep) retrieve a relevant subset of this memory and inject it into the prompt, forcing the serving engine to repeatedly prefill the same content. As the retrieval budget grows, time-to-first-token (TTFT) increases even though the underlying memory is reused across requests. We present InferScale, a GPU-native LLM memory system that replaces repeated prompt prefilling with reusable KV state. InferScale precomputes each memory fact's KV representation, stores it alongside a semantic embedding on the GPU, retrieves relevant facts at serving time, and injects their KV directly into vLLM's paged cache. To support dynamically assembled memories under rotary position embeddings, we introduce Chunked RoPE, which stores keys before rotation and applies their serving-time positions during injection. However, encoding memory facts independently omits the cross-fact context available during joint prefilling. We mitigate this with Context-Window Encoding, which encodes each memory fact together with a small window of preceding conversation context while caching only the target fact's KV. InferScale is implemented through vLLM's KV-connector interface, requiring neither engine modifications nor model fine-tuning. Across three open-weight models on LoCoMo, InferScale keeps TTFT nearly constant as the retrieval budget increases: at k=50 it reduces TTFT by 72-79% (3.6-4.8x), achieves 60.3% accuracy versus 63.3% for Mem0 without serving-time recomputation, and delivers 3.7-4.5x the throughput under concurrent load. Reusable KV state thus decouples memory-conditioned serving latency from retrieved-context size while preserving application quality.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "InferScale: GPU-Native KV Injection for Personalized LLM Serving".

Jane: Large language models are increasingly deployed with persistent personalized context, such as accumulated memory profiles or long conversation histories, that is shared across a user’s many requests.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So, let's talk about the title and who wrote this paper, InferScale: GPU-Native KV Injection for Personalized LLM Serving. Basically, the title tells us right away that they are focusing on putting Key-Value states directly into the GPU's cache instead of having to repeatedly prefill the prompt text.

Jane: That makes perfect sense when you think about how much time gets spent redoing that same initial setup for every single request; it sounds like they’re trying to eliminate that bottleneck entirely.

Lu: The authors, Peter Li and Prashant Pandey, are from Northeastern University, and their work here really bridges the gap between data systems design and direct LLM inference architecture.

Meng: I wonder how much of the complexity is baked into making it "GPU-Native" versus just being a clever software trick running on top of existing frameworks.

Lalam: For me, the fact that they are focusing on KV injection suggests a more fundamental architectural improvement in how we manage state during inference, which could lead to far more efficient and consistent personalized AI experiences for everyone.

The paper's summary: Tom: So, to summarize what InferScale actually does, the authors present a GPU-native LLM memory system that completely swaps out the old way of repeatedly prefilling prompt text with a reusable KV state. They do this by precomputing each memory fact’s Key-Value representation and injecting it directly into vLLM’s paged cache.

Jane: So, instead of the engine having to re-read and re-process that stored context every time, it's already there, ready to be used instantly when a relevant piece of memory is retrieved. That’s a significant efficiency gain for handling long conversation histories or large RAG documents.

Lu: The paper details this through two main phases: an offline phase where they build the reusable KV state, and an online serving phase that handles semantic retrieval and then injects those facts into the model's attention layer.

Meng: I see their methodology involves a clever separation between storing the semantic embedding for retrieval and storing the pre-RoPE KV representation for injection, which they link together using a shared fact id. That sounds like some pretty intricate data management to get right on hardware.

Lalam: The concept of separating the search space—the semantic embedding space for finding facts and the KV space for actual conditioning—is brilliant because it allows retrieval to happen in one area while conditioning happens in another, which is a very efficient way to handle personalization at scale.

The paper's improvements: Tom: Now, let’s look at the specific improvements they highlight. They introduce Chunked RoPE for position-independent KV reuse and Context-Window Encoding to make sure the retrieved facts are actually accurate and usable in context.

Jane: Chunked RoPE is interesting because it lets them store keys before rotary position encoding and then re-rotate them at insertion time so that they behave exactly like they were prefilled at their original spot, no matter where you inject them.

Lu: And the Context-Window Encoding is a necessary fix; it addresses the accuracy loss that happens when you encode each fact completely independently by bundling it with a small window of preceding conversation turns.

Meng: From an engineering viewpoint, the challenge they solved with Chunked RoPE is making sure that relocating those precomputed keys doesn't mess up the attention scores at all, which is a tricky thing to get right when dealing with positional encodings.

Lalam: The Context-Window Encoding ensures that even though we are retrieving facts separately, we aren't losing the surrounding conversational context needed for disambiguation; this makes the retrieved memory much more reliable for use.

Conclusion: Tom: So, wrapping things up on this InferScale paper, it really boils down to proving that retrieved memory doesn't need to be re-prefilled every time; its KV representation can be injected directly into the attention cache, which is an exact equivalence to prompt injection for any positional encoding scheme.

Jane: That means we get near-constant time-to-first-token latency regardless of how large the retrieved context becomes, which is a huge win for real applications where context size varies wildly.

Lu: The implications are quite broad; this architecture supports dynamically assembling complex, non-contiguous context blocks from retrieved facts and injecting them precisely where they are needed in the prompt.

Meng: Practically speaking, keeping all the retrieval indices and KV stores on the GPU minimizes those slow host-to-device PCIe transfers, which is essential for handling high concurrency without massive latency spikes.

Lalam: I think this architecture fundamentally enhances how we design personalized AI; it moves us toward systems where memory management isn't a performance constraint but an integrated part of the inference process.

Tom: What we've seen here with InferScale is that sophisticated memory handling can lead to much more responsive and scalable personalized AI serving. We’re really looking forward to seeing how this kind of direct KV injection plays out in production environments.

More episodes

← Home