When KV Meets Embeddings: Dynamic GPU Memory Allocation for Accelerating Generative Recommender Serving

arXiv:2605.04450 · cs.DC, cs.IR, cs.LG · Submitted 2026-08-24 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "When KV Meets Embeddings: Dynamic GPU Memory Allocation for Accelerating Generative Recommender Serving".

Jane: The paper was written by Wenjun Yu, Shuguang Han and Amelie Chi Zhou from Hong Kong Baptist University and Alibaba Inc.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper discussion segment 2: Tom: Welcome back to our discussion on "When KV Meets Embeddings: Dynamic GPU Memory Allocation for Accelerating Generative Recommender Serving." Last time, we focused on the shift from passive accounting to active resource management, discussing how the system monitors data relevance. Today, we’re going deeper into the practical mechanisms that make this dynamic allocation possible.

Jane: The core idea is implementing a sophisticated feedback loop within the model's operation. It must predict not just *if* memory is needed, but *how much* will be needed at any given moment, coordinating between the embeddings and the K/V caches seamlessly.

Meng: This requires building a predictive component that estimates data utility in real-time. Instead of just reacting to an error—like running out of memory—the system anticipates the need for more resources based on how useful the current input segments are likely to be for future predictions.

Lu: Essentially, we are creating a sophisticated resource scheduler that operates *within* the model's execution graph itself. It monitors the predicted decay rate of information relevance and allocates memory dynamically based on that observed utility decay curve.

Lalam: And this moves beyond simply optimizing hardware usage; it optimizes the *information flow*. The system learns which parts of the input context are redundant or less impactful, allowing it to aggressively prune those components while keeping the critical predictive elements available when they are needed most.

Tom: So, we are discussing the precise components—the blueprints—that allow the system to monitor usage and dynamically adjust compression and offloading based on predicted utility. This precision is what makes the whole system viable for real-world deployment. Next, we’ll look at how these improvements translate into specific, measurable gains in performance and efficiency.

Paper discussion segment 3: Tom: To recap, we’ve established that this system moves beyond simply allocating memory; it intelligently predicts data utility to manage the GPU resources required for generating complex text. Now, let's explore what this sophisticated resource choreography means for the *actual* performance and capabilities of generative models.

Jane: If we look at the hardware limitations purely through an efficiency lens, this architecture guarantees a new level of latency predictability. Previously, running a model was like driving in rush hour—your speed depended entirely on unpredictable congestion spikes. With this dynamic allocation, the system is designed to minimize those unexpected stalls by guaranteeing that critical data pathways remain clear and optimized moment by moment.

Meng: From an algorithmic standpoint, what’s revolutionary is that it decouples model capability from sheer hardware scale. Instead of needing a trillion-parameter behemoth housed in a bespoke climate-controlled facility, we can achieve similar or superior performance using smaller, more resource-aware models running on more accessible hardware. It changes the equation from ‘bigger is better’ to ‘smarter management is better.’

Lu: This architectural shift also profoundly impacts throughput. Because the system efficiently recycles memory—meaning it doesn't waste cycles flushing and reloading massive blocks of data unnecessarily—you can run multiple inference tasks concurrently on a single GPU much more effectively than before. It allows for a higher density of usage without sacrificing individual task quality.

Lalam: And this density is critical when we move beyond just text generation. Imagine combining text prompts with visual inputs, or even interpreting complex medical scans. These multimodal tasks create massive, diverse data streams that quickly overwhelm traditional memory models. This paper provides the necessary flexible framework to integrate these disparate data types without crashing the system due to resource exhaustion.

Tom: So, we are looking at a system that not only saves memory but guarantees speed and adaptability across different kinds of complex inputs—text, images, even structured databases. These aren't just marginal improvements; they represent foundational shifts in how we design AI services. This capability to handle diverse data streams reliably is what opens up the door for entirely new applications we haven't even considered yet. Next, we’ll examine the real-world economic implications of this technology breakthrough and what it means for the democratization of advanced AI tools.

Conclusion: Tom: So, to wrap up our deep dive, we've seen that the core innovation of this work is fundamentally about treating GPU memory as a highly flexible, predictive utility rather than a static storage pool.

Jane: It’s truly an architectural shift in how we approach running large generative models—it moves the bottleneck from raw hardware capacity to algorithmic intelligence.

Lu: From my perspective, the most significant takeaway is how this work operationalizes the concept of data utility decay, making it a repeatable principle for many different fields beyond just recommendation.

Meng: It highlights that the future isn't about building bigger chips; it’s about building smarter software layers that can manage those massive data matrices with extreme precision.

Lalam: Ultimately, this greatly expands the accessibility of advanced AI, allowing complex models to function reliably on a much broader range of hardware setups than before.

Tom: It’s a beautiful example of how deep computer science principles can solve very tangible engineering hurdles in real-time AI deployment.

Jane: We really appreciate you joining us today as we unpacked the implications of "When KV Meets Embeddings: Dynamic GPU Memory Allocation for Accelerating Generative Recommender Serving."

Lu: It truly represents a turning point in the economics and scalability of large-scale generative AI services.

Meng: A powerful combination of theory meeting practical, resource-constrained hardware reality.

Lalam: We

Conclusion: Tom: To wrap up our deep dive, we’ve seen that the core innovation behind "When KV Meets Embeddings: Dynamic GPU Memory Allocation for Accelerating Generative Recommender Serving" is fundamentally about treating GPU memory as a highly flexible, predictive utility rather than a static storage pool.

Jane: It really represents an architectural shift in how we approach running large generative models—it moves the primary computational bottleneck from raw hardware capacity to algorithmic intelligence and predictive resource management.

Lu: From my perspective, the most significant takeaway is how this work operationalizes the concept of data utility decay. Making that a repeatable principle is massive; it gives us a framework applicable to optimizing memory in so many different fields beyond just recommendation systems.

Meng: It highlights that the future isn't about building exponentially bigger chips; it’s about building smarter, more sophisticated software layers that can manage those massive data matrices with extreme precision and efficiency.

Lalam: And that efficiency greatly expands the accessibility of advanced AI. This technology allows complex models to function reliably on a much broader range of hardware setups than before, democratizing access to these powerful tools.

Tom: It’s truly a beautiful example of how deep computer science principles can solve very tangible engineering hurdles in real-time AI deployment, making previously unattainable performance levels achievable in practice.

Jane: We really appreciate you joining us today as we unpacked the implications of this groundbreaking research. The insights into dynamic allocation are invaluable for anyone working with large language models today.

Lu: It truly represents a turning point not just in the technical viability, but in the economic scalability of large-scale generative AI services globally.

Meng: A powerful combination of complex theory meeting practical, resource-constrained hardware reality—and that intersection is where all the most exciting breakthroughs happen.

Lalam: We hope this discussion gives you a much clearer picture of the groundbreaking potential within this entire research area and what it means for future AI development.

Tom: And with that, we’ll wrap up our discussion for today. Thank you all so much for tuning in; join us next week when we’ll be looking at some novel approaches to federated learning and how it changes the landscape of data privacy in AI.

Wenjun Yu, Shuguang Han, Amelie Chi Zhou

Hong Kong Baptist University · Alibaba Inc

cs.DC, cs.IR, cs.LG

Submitted: 2026-08-24

Updated: 2026-08-25

Importance score: 70/100

The gist: This paper introduces HLEM, a system designed to optimize Generative Recommender (GR) inference by managing the resource contention between embedding hot caches (EMB) and KV caches within limited GPU

Key concepts

Dynamic GPU Memory Allocation
This core concept involves treating GPU memory as a flexible, predictive utility rather than a static pool. The system monitors data relevance in real-time to allocate resources precisely where and when they are needed for generative models.
Data Utility Decay
The system predicts the rate at which information relevance decreases. By monitoring this decay curve, the resource scheduler can intelligently prune redundant input components while keeping critical predictive elements available for future use.
Generative Recommender Serving
This refers to using generative models to provide recommendations. The paper applies dynamic memory techniques to accelerate this process, allowing complex tasks like combining text and visual inputs without resource exhaustion.

Terminology

Summary

This paper introduces HLEM, a system designed to optimize Generative Recommender (GR) inference by managing the resource contention between embedding hot caches (EMB) and KV caches within limited GPU HBM. As GR models leverage transformer-based sequence modeling over increasingly long user interaction histories, the competition for HBM capacity becomes a critical bottleneck that can leave significant latency improvements unrealized if managed via static allocation.

The Problem: Resource Contention in GR Serving

Generative Recommender inference consists of two tightly coupled stages: embedding table lookup and transformer block computation. In production environments, both stages require persistent GPU-resident state to maintain efficiency. The embedding hot cache (EMB) reduces frequent DRAM–HBM transfers for sparse features, while the KV cache retains precomputed attention tensors to avoid redundant recomputation. However, these two caches co-exist within the same GPU HBM and are in direct competition for limited capacity.

The paper identifies a dual-bottleneck characterization where the optimal EMB–KV memory allocation ratio is highly dynamic, shifting by up to 0.35 across different workload regimes (such as varying sequence lengths or hot-user ratios). Existing systems optimize these components in isolation, which leads to two primary issues:

** Allocating more HBM to the embedding cache reduces lookup stalls but starves the KV cache, increasing recomputation overhead. **

** Allocating more HBM to the KV cache improves reuse rates but increases embedding misses and transfer stalls. **

How it works: The HLEM Architecture

HLEM (HBM-level EMB–KV Latency Manager) addresses this through a coordinated control loop involving three main components:

  1. SmartAllocator: A node-local adaptive memory allocator that uses a lightweight RL-based controller to determine the optimal EMB–KV split. It employs a three-layer PPO-based controller consisting of a frozen base policy, online residual adapter, and burst-aware recovery controller.

  2. Memory Manager: A mechanism that applies allocation updates incrementally and safely. To avoid disruptive data movement on the critical path, it uses embedding-only boundary adjustment, which treats the KV cache as a paged block pool while managing the EMB cache as a contiguous slab, ensuring updates are metadata-only and incur zero pipeline stalls.

  3. KV–EMB-Aware Scheduler: A global request router that routes requests by jointly considering KV residency, embedding locality, and node load. This prevents routing inefficiencies that occur when nodes have heterogeneous memory allocations.

Evaluation and Results

The researchers evaluated HLEM on three production-scale datasets (Taobao, Amazon Video-Games, and Amazon Books) using a 32-node A100 cluster. The results demonstrate that HLEM reduces P99 latency by 24–38% over the best static policy and achieves high service quality with 93.5–99.6% SLO satisfaction across steady, trend, and burst workloads.

The system's effectiveness is particularly evident in its ability to handle burst events, where sudden spikes in hot-user requests would typically cause static allocation policies to suffer significant latency violations. HLEM’s Recovery Controller allows it to maintain stability by shifting memory toward the KV cache during these periods, ensuring that the system remains robust against workload volatility without sacrificing throughput or incurring significant computational overhead. The decision latency for the allocator is remarkably low, recorded at only 32 µs.

Improvements for AI systems

To improve large-scale Generative Recommender (GR) serving systems, I would implement a unified management layer based on the HLEM architecture described in the paper.

The following are the specific technical improvements and their resulting capabilities:

  1. Implement a Dual-Loop RL-based HBM Partitioning Controller

The system will replace static memory allocation with a two-loop controller (SmartAllocator) using Proximal Policy Optimization (PPO) combined with an online residual adapter.

  • The primary loop uses an offline-trained PPO agent to manage the long-horizon trade-off between embedding hot caches and KV caches.

  • The secondary loop uses a lightweight residual model to adapt to real-time workload shifts (e.g., sudden spikes in hot users) and a Recovery Controller to prioritize stability during bursts.

  • This allows the system to dynamically adjust the EMB–KV HBM allocation ratio (which can shift by up to 0.35) in response to changing user interaction sequence lengths and hot-user ratios without requiring manual tuning or offline re-profiling.

  1. Deploy a Non-Disruptive Metadata-Driven Memory Manager

The system will move away from traditional memory reallocation that requires expensive data movement (cudaMemcpy) on the critical path.

  • It will implement a Zero-copy boundary adjustment mechanism where the embedding cache is managed as a contiguous LRU slab and the KV cache is managed via a paged block pool (similar to vLLM).

  • Memory resizing will be performed via lightweight metadata updates (alloc page/free page) that take microsecond-scale time.

  • Background refill of newly allocated capacity will be executed on high-priority, asynchronous CUDA streams with strict bandwidth throttling to prevent interference with latency-critical embedding misses.

  1. Integrate an EMB–KV–Load Aware Request Router

The routing layer will be upgraded from simple KV or embedding locality models to a multi-objective cost-based scheduler.

  • The router will calculate a weighted routing score for each node:

sccore(n) = wkv · hkv + wemb · hemb + wld · (1 − l(n)) + ε

where the weights are derived from actual profiled miss costs (KV recomputation vs. H2D embedding transfer).

  • The router will maintain real-time residency metadata: a user-level hash map for KV state and a shard-level set for embedding locality.

  • This allows the system to route requests to nodes that maximize the probability of both KV hits (avoiding recomputation) and EMB hits (avoiding PCIe transfers), while simultaneously balancing node load.

By implementing these three improvements, the improved AI serving system can:

  • Reduce P99 tail latency by 24–38% compared to optimized static allocation policies.

  • Achieve >93% Service Level Objective (SLO) satisfaction (e.g., <30ms latency) even during highly volatile Burst workload regimes where hot-user ratios spike suddenly.

  • Scale throughput near-linearly as the cluster size increases, effectively managing the resource contention inherent in massive embedding tables (5TB+) and long interaction sequences (15k+ tokens).

Sources

Related papers