DLB: Distributed Load Balancing at Scale for Generative AI Inference

arXiv:2609.21079 · cs.DC, cs.SY, eess.SY · Submitted 2026-09-17 · Read on arXiv

cs.DC, cs.SY, eess.SY

Submitted: 2026-09-17

Updated: 2026-09-17

Code: https://github.com/vllm-project/aibrix

Terminology

• Decode — 13× in this paper · explained in 2 episodes
• Lyapunov function — 8× in this paper · explained in 2 episodes
• Robustness — 5× in this paper · explained in 5 episodes
• Generative AI — 4× in this paper · explained in 2 episodes
• Fidelity — 3× in this paper · explained in 2 episodes
• Adaptability — 2× in this paper · explained in 1 episode
• Core — 2× in this paper · explained in 2 episodes
• Resource Constraints — 2× in this paper · explained in 3 episodes
• Speculative Decoding — 2× in this paper · explained in 2 episodes
• Transformer — 2× in this paper · explained in 2 episodes
• Adaptation — 1× in this paper · explained in 1 episode
• Additive Decomposition — 1× in this paper · explained in 1 episode
• Admissibility — 1× in this paper · explained in 2 episodes
• Benchmark — 1× in this paper · explained in 2 episodes
• Generative Model — 1× in this paper · explained in 2 episodes
• Generative Models — 1× in this paper · explained in 2 episodes
• Homogeneity — 1× in this paper · explained in 2 episodes
• Introspection — 1× in this paper · explained in 2 episodes
• KV cache — 1× in this paper · explained in 3 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Nash Equilibrium — 1× in this paper · explained in 2 episodes
• Objective Function — 1× in this paper · explained in 2 episodes
• Redundancy — 1× in this paper · explained in 2 episodes
• Theoretical Guarantees — 1× in this paper · explained in 2 episodes
• Trace — 1× in this paper · explained in 2 episodes

Sources

• Load Balancing with Network Latencies via Distributed Gradient Descent
• GORGO: Online Tuning for Cross-Region Network-Aware LLM Serving
• SkyWalker: A Locality-Aware Cross-Region Load Balancer for LLM Inference
• DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Serving

Related papers

• iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
• SAMM: Sharded Automated Market Maker
• InferScale: GPU-Native KV Injection for Personalized LLM Serving
• Vigil: Accountable Liveness against Selective Silence
• Steelhead: Interleaving Partially Synchronous and Asynchronous Commit Rules on a Shared DAG
• Pushing CPU Speech Synthesis to the Wall: Extreme Inference Tuning under Serverless Architecture and Billing