OctoPipe: Reducing Pipeline Bubbles for Heterogeneous Models via Co-Optimizing Partitioning, Placement, and Scheduling

arXiv:2509.23722 · cs.DC, cs.AI · Submitted 2025-09-28 · Read on arXiv

cs.DC, cs.AI

Submitted: 2025-09-28

Updated: 2026-09-02

Code: https://github.com/NVIDIA/Megatron-LM

Terminology

• Pipeline Parallelism — 23× in this paper
• Transformer — 6× in this paper · explained in 2 episodes
• Trace — 5× in this paper · explained in 2 episodes
• Fidelity — 4× in this paper · explained in 2 episodes
• Accuracy — 2× in this paper · explained in 1 episode
• Communication Overhead — 2× in this paper · explained in 2 episodes
• Foundation Model — 2× in this paper · explained in 3 episodes
• Foundation Models — 2× in this paper · explained in 7 episodes
• Large Language Models (LLMs) — 2× in this paper · explained in 6 episodes
• Transformer Models — 2× in this paper · explained in 2 episodes
• Abstraction — 1× in this paper · explained in 1 episode
• Activation Checkpointing — 1× in this paper · explained in 1 episode
• Adaptation — 1× in this paper · explained in 1 episode
• Adaptive Partitioning — 1× in this paper · explained in 1 episode
• Data Heterogeneity — 1× in this paper · explained in 3 episodes
• Data Parallelism — 1× in this paper
• Directed Acyclic Graph — 1× in this paper
• Feed-Forward Network — 1× in this paper
• Integer Linear Programming — 1× in this paper
• Resource Constraints — 1× in this paper · explained in 3 episodes
• Robustness — 1× in this paper · explained in 5 episodes
• SMT solver — 1× in this paper · explained in 2 episodes

Sources

• The Llama 3 Herd of Models
• Kimi K2: Open Agentic Intelligence
• gpt-oss-120b & gpt-oss-20b Model Card
• Qwen3 Technical Report
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model
• MiniMax-01: Scaling Foundation Models with Lightning Attention
• Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Horovod: fast and easy distributed deep learning in TensorFlow
• DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models
• Training Deep Nets with Sublinear Memory Cost
• SPPO:Efficient Long-sequence LLM Training via Adaptive Sequence Pipeline Parallel Offloading

Related papers

• iScheduler: Reinforcement Learning-Driven Continual Optimization for Large-Scale Resource Investment Problems
• SAMM: Sharded Automated Market Maker
• InferScale: GPU-Native KV Injection for Personalized LLM Serving
• Vigil: Accountable Liveness against Selective Silence
• Steelhead: Interleaving Partially Synchronous and Asynchronous Commit Rules on a Shared DAG
• Pushing CPU Speech Synthesis to the Wall: Extreme Inference Tuning under Serverless Architecture and Billing