ImpactHO: Importance-Aware KV Cache Transfer for Multi-User Edge LLM Handover

arXiv: 2608.10545v1 [cs.NI] 11 Aug 2026

---

Edge LLMs must preserve inference continuity when a user hands over between edge nodes, requiring key-value (KV) cache transfer to the target node. However, simultaneous handovers saturate the backhaul, preventing full cache delivery within the mobility-imposed transfer window. The paper states: "Rather than allocating bandwidth as if all cache entries were equally valuable, we order each user's KV cache by importance and transmit only its most informative fraction, turning token-level sparsity into communication savings."

The authors note that the KV cache is substantially larger than the raw text it encodes, so concurrent long-context handovers still saturate the backhaul. They also observe that modern AI models carry tens to hundreds of gigabytes of weights (e.g., 16 GB for 8B-parameter models), making such migration impractical and pushing handover latency to tens of seconds.

The paper proposes ImpactHO (Importance-aware KV cache transfer for multi-user handover), a framework that orders each user's cache by importance and transmits its most informative entries first. The scheduling task is formulated as a utility-maximization problem: allocate the limited backhaul bandwidth across users so as to maximize the average accuracy.

The framework casts the transfer as a multi-user backhaul allocation problem that maximizes average accuracy across users. Each user's partial-cache accuracy serves as its utility: "a sigmoid that fits measurements on the RULER benchmark with R2 > 0.99 across models and context lengths."

1. ImpactHO framework: We formulate importance-ordered partial KV cache transfer as a multi-user backhaul allocation problem for edge LLM handover, repurposing per-entry importance scores from KV cache eviction to set the transmission order.

2. Empirical sigmoid characterization: "On the RULER benchmark, partial-cache accuracy follows a sigmoid (R2 > 0.99) robustly across context lengths, models, and ordering schemes. Importance ordering pulls its inflection point down to about 6.5% of the cache, so the concave region spans nearly the entire cache."

3. Two-regime allocator with low overhead: "We derive the per-slot optimum in closed form as a weighted water-filling solution over a feasible region expanded by importance ordering. It reduces to classical water-filling as a special case and runs in near-linear time per slot."

The KV cache size for user i is given by:

Li = 2nL nH dh qTi bits, where nL is the number of transformer layers, nH is the number of KV heads, dh is the head dimension, q is the number of bits per scalar, and Ti is the context length.

For example, the KV cache of Qwen3-8B occupies approximately 1.2 GB (9.66 Gb) for an 8K-token context.

The system uses discrete time slots of duration Δt. At the beginning of each slot, the scheduler observes the set of active handover users and determines their allocations, which remain fixed throughout the slot.

The paper models per-user utility Ai(y) as the inference accuracy for user i when the target node has received a fraction y ∈ [0,1] of the user's importance-ordered KV cache. The algebraic sigmoid form is adopted:

Ai(y) = (Mi/2) [1 + ki(y − τi) / sqrt(1 + ki2(y − τi)2)]

with marginal utility:

A′i(y) = Mi ki / [2 (1 + ki2(y − τi)2)(3/2)]

The parameters Mi, ki, and τi represent the upper accuracy asymptote, transition sharpness, and concavity anchor, respectively.

Empirical results show: "The algebraic sigmoid remains accurate across the 4K, 8K, and 16K context lengths, achieving R2 > 0.999 in every case. The fitted inflection points lie within the narrow range τ ∈ [0.064, 0.067]."

The per-slot allocation problem is formulated as:

maximize Σi [Ai(yi) − Ai(xi)] subject to Σi Li bi ≤ B, bi ≥ 0, τi ≤ yi ≤ 1

where xi is the received fraction at slot start, yi = xi + biΔt is the received fraction at slot end, and B is the backhaul bandwidth.

The feasibility condition (Lemma 1) requires:

B ≥ Bmin = (1/Δt) Σi Li[τi − xi]+

Theorem 1 (Weighted Water-Filling): Under Assumption 1, if B ≥ Bmin, the primal optimum is:

**yi⋆ = max xi, Wi(λ⋆) **, b⋆i = (yi⋆ − xi)/Δt

where Wi(λ) is defined piecewise with three branches (interior, upper bound, lower bound), and λ⋆ is a dual-optimal variable satisfying the budget constraint.

The paper notes: The optimal allocation equalizes the marginal accuracy gain per additional transmitted cache bit across all interior users.

For the algebraic sigmoid, the interior branch gives:

(A′i)−1(si(λ)) = τi + (1/ki) sqrt((Mi ki / (2 si(λ)))(2/3) − 1)

When B < Bmin, the scheduler invokes an admission policy called Equalized Bytes (EB): EB equalizes the number of cache bits delivered during the slot among the users in S (users that have not yet reached their concavity anchors). By prioritizing users that have not yet reached their anchors, EB mitigates sub-τ starvation under heavy load.

• At the default ρ = 4 users/s, it attains 93.7% accuracy, within 0.5 pp of the 94.1% full-cache ceiling.

• The proposed allocator attains 98.2–99.5% of a clairvoyant upper bound across all loads.

• The proposed allocator consistently outperforms all three baselines across the entire operating range when sweeping backhaul bandwidth B over 10–30 Gbps.

• The proposed allocator stays above all three baselines across the entire range when sweeping slot duration Δt over 20–100 ms.

• When sweeping transfer window Tmax over 200–1000 ms, the proposed allocator again outperforms throughout.

• Importance ordering: Importance ordering keeps average accuracy above 90% across the entire range, whereas random ordering falls sharply from 56.7% at ρ = 2 to 20.2% at ρ = 8.

• Admission control: At ρ = 12 users/s, the sub-τ starvation rate is 20.8% without admission versus 0.65% with EB. EB consistently outperforms WTA, whose starvation rate is 15.5% versus EB's 0.65%.

The proposed method achieves average latency of 391 ms versus 847 ms for re-prefill and 434 ms for the hybrid approach. As the context length increases, the proposed method outperforms the baselines, which suffer from severe computational overhead during full context recomputation.

The paper identifies three source-side costs: (i) Scoring via Fast KVzip is a one-time, per-session computation done while the source still serves the user, not at handover; (ii) Sorting is an O(n log n) sort over indices, a small fraction of the transfer time; (iii) Metadata overhead is under 1% overhead (3-byte coordinates against 0.5 KB payloads).

The paper concludes: These results confirm that jointly accounting for KV cache importance and backhaul resource allocation enables accurate and efficient LLM handover at the network edge. The framework combines "importance-ordered sequential KV cache transfer, an empirically validated sigmoid characterization of inference accuracy with partially transferred caches, and an optimal weighted water-filling allocator whose homogeneous special case reduces to classical water-filling."

Based on this paper, I can improve AI systems in the following specific ways:

1. Implement importance-aware partial context transfer for distributed inference

• Improved AI system: An LLM serving system that, when migrating a conversation between compute nodes, transmits only the most critical KV cache entries (identified via importance scores from eviction algorithms) rather than the full cache. This reduces handover latency from 847ms to 391ms and maintains 93.7% accuracy (within 0.5pp of full-cache ceiling) even under multi-user backhaul constraints.

2. Add adaptive accuracy-aware bandwidth allocation across concurrent users

• Improved AI system: A multi-tenant LLM inference orchestrator that, when multiple users hand over simultaneously, allocates limited backhaul bandwidth using the weighted water-filling solution (Theorem 1). This maximizes average inference accuracy across all users, achieving 98.2–99.5% of a clairvoyant upper bound, versus naive equal-split or priority-based schemes that degrade sharply under load.

3. Embed a sigmoid accuracy model for partial-context inference

• Improved AI system: An LLM runtime that predicts inference quality as a function of the fraction of KV cache received, using the fitted algebraic sigmoid (R2 > 0.99). This enables the system to make real-time decisions—e.g., whether to wait for more cache, start inference with partial context, or trigger fallback—based on a closed-form marginal utility, rather than heuristics or full recomputation.

4. Incorporate admission control to prevent starvation under overload

• Improved AI system: An edge LLM gateway that, when backhaul is insufficient to deliver even the minimum required cache fraction (τ ≈ 6.5%) to all users, uses the Equalized Bytes policy to prioritize users below their concavity anchor. This reduces sub-τ starvation from 20.8% (no admission) to 0.65%, ensuring no user gets unusably incomplete context during handover spikes.

5. Enable token-level importance scoring as a first-class system primitive

• Improved AI system: An LLM inference engine that computes per-token KV importance scores during normal serving (not at handover time) and stores them as lightweight metadata (3-byte coordinates per entry, <1% overhead). This makes importance-ordered transfer, eviction, and compression interoperable, allowing the same scores to drive both memory management and network-efficient migration.

6. Provide a closed-form, near-linear-time scheduler for dynamic handover

• Improved AI system: A real-time scheduler that, at each time slot (20–100ms), observes active handover users and computes optimal per-user bit allocations in near-linear time via the three-branch weighted water-filling formula. This is suitable for edge deployments with hundreds of concurrent handovers, where iterative optimization would be too slow.