Reforge: Low-Latency Distributed GNN Serving with Selective Embedding Recomputation

arXiv:2501.08547 · cs.DC, cs.LG · Submitted 2025-01-15 · Read on arXiv

cs.DC, cs.LG

Submitted: 2025-01-15

Updated: 2026-09-21

Comments: Extended version of the IPDPS'26 paper (https://doi.org/10.1109/IPDPS65963.2026.00071)

Code: https://github.com/NVIDIA/nccl

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Accuracy — 88× in this paper · explained in 1 episode
• Embeddings — 68× in this paper · explained in 2 episodes
• Aggregation — 57× in this paper · explained in 1 episode
• Communication Overhead — 16× in this paper · explained in 2 episodes
• GNNs — 12× in this paper · explained in 2 episodes
• Graph Convolutional Networks — 9× in this paper
• Historical Embeddings — 9× in this paper
• Computation Graph Parallelism — 7× in this paper
• Representation Learning — 6× in this paper · explained in 2 episodes
• Aggregation Function — 4× in this paper · explained in 1 episode
• Benchmark — 4× in this paper · explained in 2 episodes
• Core — 4× in this paper · explained in 2 episodes
• Selective Recomputation of Precomputed Embeddings — 4× in this paper
• Deep Graph Library — 3× in this paper
• Graph Neural Networks (GNNs) — 3× in this paper · explained in 7 episodes
• Neighborhood Sampling — 3× in this paper
• Transformer — 3× in this paper · explained in 2 episodes
• Adam — 2× in this paper · explained in 2 episodes
• Attention Mechanism — 2× in this paper · explained in 2 episodes
• Over-smoothing — 2× in this paper · explained in 4 episodes
• Redundancy — 2× in this paper · explained in 2 episodes
• Adaptive Graph — 1× in this paper · explained in 1 episode
• Adaptive Sampling — 1× in this paper · explained in 1 episode
• Adjacency Matrix — 1× in this paper · explained in 1 episode
• Computational overhead — 1× in this paper · explained in 2 episodes
• Contrastive Learning — 1× in this paper · explained in 5 episodes
• Distillation — 1× in this paper · explained in 2 episodes
• Feature Extraction — 1× in this paper · explained in 2 episodes
• Graph Construction — 1× in this paper · explained in 3 episodes
• Heterogeneous Graph — 1× in this paper · explained in 2 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Loss Function — 1× in this paper · explained in 2 episodes
• Semi-supervised Learning — 1× in this paper · explained in 4 episodes
• Trace — 1× in this paper · explained in 2 episodes
• Transformer Models — 1× in this paper · explained in 2 episodes

Sources

• GPT-4 Technical Report
• How Attentive are Graph Attention Networks?
• Representation Learning on Graphs: Methods and Applications
• Training Compute-Optimal Large Language Models
• Adam: A Method for Stochastic Optimization
• DeeperGCN: All You Need to Train Deeper GCNs
• Quiver: Supporting GPUs for Low-Latency, High-Throughput GNN Serving with Workload Awareness
• Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks

Related papers

• iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
• SAMM: Sharded Automated Market Maker
• InferScale: GPU-Native KV Injection for Personalized LLM Serving
• Vigil: Accountable Liveness against Selective Silence
• Steelhead: Interleaving Partially Synchronous and Asynchronous Commit Rules on a Shared DAG
• Pushing CPU Speech Synthesis to the Wall: Extreme Inference Tuning under Serverless Architecture and Billing