SlideDP: Scaling Host-Resident LLM Fine-Tuning Across Multiple GPUs

summary

Video file (mp4)

The gist

The gist The paper presents SlideDP, a synchronous data-parallel runtime for shared-host multi-GPU systems, which enables full-parameter LLM fine-tuning beyond GPU memory by coordinating parameter

In short

SlideDP is a synchronous runtime for fine-tuning large language models across shared-host multi-GPU systems. It enables training beyond GPU memory by coordinating parameter delivery, gradient aggregation, and CPU updates across ranks. The system uses an analytical model and policy selection to optimize communication routes and activation placement based on hardware topology.

Key concepts

One Authoritative Host State
SlideDP maintains a single, definitive version of the host state. This state is decoupled from the specific layout used for parameters or gradients. This separation allows the system to choose different communication routes based on the current workload and hardware topology without disrupting the central, authoritative host view.
Elastic Checkpointing
This mechanism defines how activations are saved, allowing flexible control over memory usage. It includes policies for checkpoint granularity and an offload ratio that dictates where activations are placed—either in GPU memory or on the host CPU—to manage the trade-off between speed and memory capacity.
Analytical Step-Time Model
This model analyzes how resources bottleneck during shared-host scaling. It explains how reducing data traffic can shorten training steps, and when shrinking computation windows exposes work on the host. This informs the runtime's decisions about communication routes and chunk sizes to maintain optimal pipeline exposure.
Marginal Pipeline Cost (τj)
This calculated coefficient measures the marginal cost of a specific communication route or configuration. It captures the overlap and contention between different operations. The system uses this metric to select policies that minimize the total cost, ensuring a memory-validated policy for fine-tuning.

Terminology used across episodes

This episode discusses

The paper

SlideDP: Scaling Host-Resident LLM Fine-Tuning Across Multiple GPUs · Read on arXiv

Ruijia Yang, Shiyuan Lin, Yulong Ao, Zhiyu Li, Yingli Zhao, Xianduo Li, Yonghua Lin

The Hong Kong University of Science and Technology (Guangzhou) · Beijing Academy of Artificial Intelligence

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "SlideDP: Scaling Host-Resident LLM Fine-Tuning Across Multiple GPUs".

Jane: The gist The paper presents SlideDP, a synchronous data-parallel runtime for shared-host multi-GPU systems, which enables full-parameter LLM fine-tuning beyond GPU memory by coordinating parameter delivery, gradient aggregation,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper called "SlideDP: Scaling Host-Resident LLM Fine-Tuning Across Multiple GPUs." Basically, they're tackling the problem of fine-tuning massive AI models that are too big to fit entirely on the GPU memory.

Jane: That’s right. The main idea is that it enables full-parameter fine-tuning beyond what a single GPU can hold by coordinating things across multiple GPUs when they share a host machine.

Lu: It claims this synchronous data-parallel runtime maintains one authoritative host state while decoupling the communication routes from the actual layout and pipelines parameter delivery, gradient aggregation, and CPU updates across all those ranks <ref:2609.34162#pg1>.

Meng: So it’s not just about moving data; it’s about orchestrating the whole process so that even with shared host resources, you can still train these huge models efficiently.

Lalam: It essentially sets up a system that manages parameter delivery, gradient return, and layer-wise CPU updates all within a bounded GPU working window <ref:2609.34162#pg2>.

Tom: And they use an analytical step-time model to characterize the resource bottlenecks and pipeline exposure, which helps them guide the communication routes and chunk sizes based on topology and workload <ref:2609.34162#pg1>.

Jane: It’s about figuring out when reducing traffic shortens a step, or when shrinking computation windows exposes host work, which is something they call residual pipeline exposure <ref:2609.34162#pg2>.

Meng: So the core challenge they're addressing seems to be how to manage that remaining shared-host work so it overlaps with the GPU compute instead of just sitting there and wasting time.

Lu: They use something called AutoPolicy which configures communication routes, chunk sizes, and activation layouts for the specific workload and hardware you're using <ref:2609.34162#pg1>.

Tom: And this policy selection involves a measured difference, tau j equals tau zero plus T j k j minus T zero k j, which captures that marginal pipeline cost of overlap and contention <ref:2609.34162#pg3>.

Jane: They then minimize the sum over all candidate layers n j times tau j equals L to find a memory-validated policy called policy pi <ref:2609.34162#pg3>.

Tom: In terms of performance, in matchedbatch sweeps, SlideDP achieves geometric-mean throughput ratios of one point four six to two point six four times over SlideFormer and MegaTrain <ref:2609.34162#pg1>.

Meng: That’s a solid number. And they report that on four H100s, SlideDP approaches GPU-resident FSDP2 throughput for Qwen3-14B even at a smaller batch size <ref:2609.34162#pg1>.

Jane: They also mention that with a larger batch, it processes over 1M tokens per step and exceeds FSDP2’s measured peak throughput by eleven point two percent <ref:2609.34162#pg1>.

Lu: Furthermore, they sustain geometric-mean speedups of one point eight three times over ZeRO-Offload and one point four eight times over MegaTrain across batches four to thirty-two on four RTX four thousand ninety GPUs <ref:2609.34162#pg1>.

Tom: And they note that this system reduces Qwen3-14B peak reserved memory by fifty-one to fifty-nine percent versus ZeRO-Offload and twenty-nine to forty-two percent versus MegaTrain at common feasible batches <ref:2609.34162#pg1>.

Jane: It’s clear that this paper, "SlideDP: Scaling Host-Resident LLM Fine-Tuning Across Multiple GPUs," shows that shared host DP requires coordinated decisions across state movement, execution scheduling, and memory allocation rather than just optimizing each part separately <ref:2609.34162#pg1>.

Meng: So for a practitioner looking at this, the implication is that you can translate host-memory capacity into larger feasible training workloads across commodity workstations and multi-GPU servers <ref:2609.34162#pg1>.

Lu: Observation one points out an avoidable amplification where direct DP extension causes host demand to grow with GPU rank count while the host service capacity stays fixed <ref:2609.34162#pg2>.

Tom: That’s a big warning there. Then they address observation two, residual pipeline exposure, which is when host-side work gets exposed when its completion exceeds the hiding window before downstream operations need that result <ref:2609.34162#pg2>.

Jane: And observation three shows that the right delivery route really depends on both topology and workload; sharded delivery improves throughput by sixteen point eight to thirty-two point seven percent with NVLink bridges compared to replicated delivery <ref:2609.34162#pg2>.

Meng: Observation four highlights that the value of spare HBM depends on exposed pipeline work, suggesting memory allocation should be a runtime policy decision rather than a fixed checkpointing choice <ref:2609.34162#pg1>.

Lu: So SlideDP shows that you need coordinated decisions across state movement, execution scheduling, and memory allocation instead of optimizing individual components independently <ref:2609.34162#pg1>.

Tom: It’s about using the host-memory capacity to support larger training workloads while maintaining throughput across a whole range of hardware setups <ref:2609.34162#pg1>.

Jane: This paper, "SlideDP: Scaling Host-Resident LLM Fine-Tuning Across Multiple GPUs," is showing us how to make these massive models more accessible by coordinating the host resources much more intelligently <ref:2609.34162#pg1>.

Conclusion: Tom: So we're wrapping up on SlideDP, which is about scaling those huge LLMs when they don't fit on a single GPU by coordinating everything across multiple GPUs sharing one host machine <ref:2609.34162#pg1>.

Jane: It’s really about making that shared-host setup actually work for massive fine-tuning tasks, not just fitting the model in there.

Lu: The authors are SlideDP, and they're basically showing how to manage parameter delivery and gradient aggregation across ranks while keeping one main host state consistent <ref:2609.34162#pg1>.

Meng: So what’s the big picture here for someone actually building these systems?

Tom: It means we can use more HBM because you can move the host memory capacity into a much larger feasible training workload, which is huge for scaling up model size.

Jane: They found that this coordination works really well across different hardware, even on commodity workstations and multi-GPU servers <ref:2609.34162#pg1>.

Lu: Their policy selection method uses an analytical step-time model to figure out the best way to route data based on the topology and the specific workload you're running <ref:2609.34162#pg1>.

Meng: That sounds complex. What’s the practical win beyond just getting a better throughput number?

Tom: The real win is that they identified issues like avoidable amplification and residual pipeline exposure, telling us exactly where we need to tune things in the software stack <ref:2609.34162#pg2>.

Jane: And they pointed out that memory allocation should be a runtime decision based on how much work is exposed in the pipeline, not just a fixed checkpointing choice <ref:2609.34162#pg1>.

Lu: It suggests we need a more holistic approach to managing shared-host DP where state movement, execution scheduling, and memory allocation are all coordinated <ref:2609.34162#pg1>.

Meng: So it’s about coordination instead of just optimizing each component in isolation.

Tom: Exactly. SlideDP shows that you can use host-memory capacity to support much larger training jobs while still maintaining decent throughput across a variety of setups <ref:2609.34162#pg1>.

Jane: It’s about making these massive models more accessible by coordinating the host resources much more intelligently <ref:2609.34162#pg1>.

Tom: Next up, we'll look at some of the specific performance numbers they got from those experiments and see how they stack up against other methods.

More episodes

← Home