Pipeline Parallelism
Pipeline parallelism splits a model that is too large for the memory of a single process across its layers, and gives each process a subset of layers, called a stage, as in the stages of a pipeline. The forward and backward passes then communicate activations and gradients between connecting stages, as illustrated below:
Training of a model split into 4 stages on 4 GPUs, without pipelining: the forward pass of a mini-batch goes from GPU 0 to GPU 3, and its backward pass, which takes twice as long, comes back, so only one GPU computes at a time. The number in each cell is the mini-batch index. Adapted from: PipeDream: Generalized Pipeline Parallelism for DNN Training
Note that this method yields a low utilization of compute resources, as each stage waits for the connecting stages before it computes its share of the forward and backward passes. In the picture above, each GPU idles for three quarters of the time.
Pipeline parallelism can be combined with other types of parallelism, such as data parallelism and tensor parallelism. With data parallelism, the GPUs are split into several pipelines, each holding a copy of the model and processing its own share of the mini-batch. At the end of each mini-batch, the GPUs that hold the same stage average their gradients with an all-reduce. In practice, the number of stages is the smallest one that fits the model in memory, as fewer stages mean less idle time. However, no combination removes the idle time of a pipeline.
Micro-batching
Micro-batching reduces this idle time: the mini-batch is split into \(m\) micro-batches that enter the pipeline one after the other, so that the stages work on different micro-batches at the same time. The gradients of the micro-batches are added, and the optimizer updates the parameters once all backward passes are complete, i.e. when the pipeline is empty (a pipeline flush).
In practice, micro-batching in a pipeline is equivalent to gradient accumulation: all micro-batches use the same parameters, and the sum of their gradients, with the loss of each divided by \(m\), is the gradient of the whole mini-batch. The only difference is that the stages process different micro-batches at the same time, instead of one after the other.
The order of the forward and backward passes on each GPU is called the schedule. The simplest one, used by GPipe, runs all forward passes, then all backward passes, as illustrated below:
GPipe on 4 GPUs with 8 micro-batches: all forward passes, then all backward passes. Each GPU keeps the activations of the 8 micro-batches until their backward passes. Adapted from: Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
The schedule diagrams with 8 micro-batches were generated from the reference implementation of each schedule, on 4 GPUs (8 GPUs and 16 micro-batches for DualPipe). Those on 4 GPUs share one time scale: a forward pass takes 1 unit and a backward pass 2 units (1+1 when it is split, as in Zero Bubble), and communication time is not modeled.
The idle time that remains is called the pipeline bubble, by analogy with the bubbles of a processor pipeline. With \(p\) stages, every stage idles for the time of \(p-1\) forward and \(p-1\) backward passes in each iteration (3 of each in the picture above), whatever the value of \(m\). Therefore, the bubble fraction, i.e. the idle time relative to the compute time of an iteration, is given by:
\[\frac{(p-1)(F+B)}{m(F+B)} = \frac{p-1}{m}\]where \(F\) and \(B\) are the times of the forward and backward passes of one micro-batch on one stage. Thus, the bubble is small only when there are many more micro-batches than stages. Instead of shrinking the bubbles, PipeFill runs other jobs in them.
1F1B
The most common schedule is 1F1B (one forward, one backward). On GPU \(i\), from 0 to \(p-1\), an iteration runs in three phases. A warm-up runs the forward passes of the first \(p-i-1\) micro-batches, a steady state alternates one forward and one backward pass, and a cool-down runs the remaining backward passes. 1F1B is illustrated below, on the same time scale:
1F1B on 4 GPUs with 8 micro-batches: after a warm-up of forward passes, each GPU alternates one forward and one backward pass. The iteration takes as long as with GPipe, but each GPU keeps the activations of at most 4 micro-batches instead of 8. Adapted from: Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
1F1B takes as long as the simplest schedule, with the same bubble, but each backward pass releases the activations of its micro-batch. Thus, GPU \(i\) keeps at most \(p-i\) micro-batches in flight (whose forward pass is done and whose backward pass is not), instead of all \(m\).
1F1B is the default schedule of DeepSpeed and Megatron-LM, and the base of most schedules below. It combines two earlier methods, GPipe and PipeDream, presented next.
GPipe
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism (Google, 2018, arXiv) introduced micro-batching, with the simplest schedule shown above: all forward passes, then all backward passes. The paper found the bubble negligible with at least four times as many micro-batches as stages (\(m \geq 4p\)). The following picture illustrates GPipe with 4 micro-batches and its pipeline flush:
Pipeline parallelism with gradient accumulation (GPipe) on 4 GPUs: the mini-batch is split into 4 micro-batches, all forward passes run before the backward passes, and the gradients are added at the pipeline flush, before the next mini-batch starts. Adapted from: PipeDream: Generalized Pipeline Parallelism for DNN Training
The price of a small bubble is memory: GPipe stores the activations of all \(m\) micro-batches until their backward passes, against at most \(p\) for 1F1B. CPU offloading reduces it by moving the activations to the CPU and back, at the cost of data transfers. Activation checkpointing, which GPipe applies at the beginning of every stage, recomputes them in the backward pass instead, at the cost of extra computation.
PipeDream
PipeDream: Generalized Pipeline Parallelism for DNN Training (Microsoft, SOSP 2019) introduced the 1F1B schedule. It never flushes the pipeline, so that the GPUs do not idle between mini-batches: the forward and backward passes of different mini-batches overlap, as illustrated below:
The PipeDream schedule on 4 GPUs. In the startup state, GPU 0 admits 4 mini-batches; in the steady state, each GPU alternates the forward pass of one mini-batch and the backward pass of an earlier one, so several mini-batches are in flight at the same time. Backward passes are prioritized over forward passes on each GPU. Adapted from: PipeDream: Generalized Pipeline Parallelism for DNN Training
Where is the caveat? Each GPU updates its weights after every backward pass, so the backward pass of a mini-batch would compute its gradients with other weights than its forward pass. PipeDream therefore keeps several versions of the parameters (weight stashing): forward passes use the latest version, and the backward pass of a mini-batch uses the version of its forward pass.
Weight stashing keeps one version of the weights for each mini-batch in flight, up to 4 on GPU 0 in the picture above, which increases memory substantially.
PipeDream-Flush
Memory-Efficient Pipeline-Parallel DNN Training (Microsoft, ICML 2021, arXiv) introduced PipeDream-Flush, which keeps the 1F1B schedule of PipeDream but brings back the pipeline flush of GPipe at the end of every mini-batch, so that a single version of the weights suffices. This is the 1F1B schedule illustrated above. For the sake of comparison with GPipe, the following picture illustrates PipeDream-Flush with 4 micro-batches:
PipeDream-Flush on 4 GPUs with 4 micro-batches: after a warm-up of forward passes, each GPU alternates one forward and one backward pass, and the gradients are added at the pipeline flush, as in GPipe. The iteration takes as long as with GPipe, but GPU 3 keeps one micro-batch in flight instead of 4. Inspired by: Memory-Efficient Pipeline-Parallel DNN Training
Interleaved 1F1B
With 1F1B, the bubble fraction \((p-1)/m\) shrinks only with more micro-batches or with fewer stages. However, a larger batch is limited by the convergence of training, smaller micro-batches use the GPU less efficiently, and fewer stages need more memory per GPU.
Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM (NVIDIA, 2021, arXiv) introduced interleaved 1F1B, which gives each GPU several non-contiguous chunks of layers (virtual stages) instead of one block of consecutive layers. With 2 chunks per GPU, GPU \(i\) holds the chunks \(i\) and \(i+p\) of the \(2p\) chunks, so every micro-batch loops through the GPUs twice.
Smaller chunks make the pipeline fill and drain faster, and the bubble shrinks by the number of chunks per GPU \(v\), to \((p-1)(F+B)/v\). The price is \(v\) times more point-to-point communication, and more activation memory, as the first GPU keeps more micro-batches in flight. The original implementation also requires the number of micro-batches to be a multiple of the number of GPUs. This schedule is available in Megatron-LM and as ScheduleInterleaved1F1B in PyTorch’s torch.distributed.pipelining, and can be illustrated as:
Interleaved 1F1B with 2 model chunks per GPU. Light and dark cells refer to the first and second chunk on each GPU. Interleaving halves the bubble of 1F1B. Adapted from: Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
Zero Bubble (ZB-H1 and ZB-H2)
Zero Bubble Pipeline Parallelism (Sea AI Lab, ICLR 2024, arXiv) builds on the observation that the backward pass consists of two independent computations. For a linear layer \(y = xA\) and a loss \(L\), the gradient with respect to the input, \(\partial L / \partial x = (\partial L / \partial y) A^T\), is needed right away by the previous stage, so it is on the critical path. The gradient with respect to the weights, \(\partial L / \partial A = x^T (\partial L / \partial y)\), is only needed by the optimizer step, so it can be postponed.
Zero Bubble therefore splits every backward pass in two: the backward for inputs is computed and sent upstream immediately, and the backward for weights is deferred to fill the bubbles. The paper proposes two handcrafted schedules:
- ZB-H1 keeps the peak activation memory of 1F1B. The deferred backward for weights passes fill the end of the iteration, which reduces the bubble to \((p-1)(F+B-2W)\), where \(W\) is the time of the backward for weights (\(B\) is still the time of the whole backward pass). When the forward, backward for inputs and backward for weights take the same time, this is a third of the 1F1B bubble.
- ZB-H2 allows up to \(2p-1\) micro-batches in flight on the first stage, about twice the activation memory of 1F1B. The extra forward passes fill the warm-up bubble, and the reordered backward for weights passes turn the schedule into a parallelogram that interlocks with the previous and next iterations, so the bubble disappears.
There is a catch: the optimizer step usually synchronizes all stages, e.g. to compute the global gradient norm for gradient clipping, or to check for NaN/Inf values in mixed precision training, which breaks the parallelogram. The authors replace it with an optimizer post-validation: each stage updates its weights right away, and the rare update that triggers clipping or the NaN/Inf check is rolled back and redone.
They also provide an algorithm that searches for the best schedule, given the measured time of each pass and of the communication, and a memory limit. In their experiments, the throughput of Zero Bubble was up to 23% higher than that of 1F1B under a similar memory limit, and up to 31% higher with a relaxed one. The implementation is available as a fork of Megatron-LM and as ScheduleInterleavedZeroBubble in PyTorch. ZB-H1 and ZB-H2 are illustrated below:
ZB-H1, with the activation memory of 1F1B. Every backward pass is split into a backward for inputs and a backward for weights, and the latter fill the bubbles. Adapted from: Zero Bubble Pipeline Parallelism
ZB-H2, with about twice the activation memory of 1F1B. When the optimizer step does not synchronize the stages, the next iteration starts on the first GPUs while the last GPUs finish the current one; in the same way, the previous iteration fills the idle time at the start. Adapted from: Zero Bubble Pipeline Parallelism
ZB-V
The Zero Bubble paper also introduces ZB-V, which places two model chunks on each GPU in a V shape: GPU \(i\) holds the chunks \(i\) and \(2p-1-i\) of the \(2p\) chunks. A micro-batch goes down the GPUs through the first half of the model and comes back up through the second half, so the first and last chunks (embeddings and loss) are both on GPU 0, and the turn on the last GPU needs no communication.
When the forward, backward for inputs and backward for weights take the same time, ZB-V has zero bubble (with the optimizer post-validation) and the peak activation memory of 1F1B, balanced across all GPUs. The price is twice the point-to-point communication of 1F1B. ZB-V is available as ScheduleZBVZeroBubble in PyTorch, and is illustrated below:
ZB-V with 8 model chunks on 4 GPUs. Light cells refer to the first chunk on each GPU (chunks 0 to 3, going down) and dark cells to the second chunk (chunks 4 to 7, going back up). As in ZB-H2, the previous iteration fills the idle time at the start when the optimizer step does not synchronize the stages. Adapted from: Zero Bubble Pipeline Parallelism
Going further, Pipeline Parallelism with Controllable Memory (Sea AI Lab, 2024, arXiv) generalizes V-shape schedules to trade bubbles for memory, down to half (V-Half) and a third (V-Min) of the activation memory of 1F1B.
DualPipe
DeepSeek-V3 Technical Report (DeepSeek-AI, 2024, arXiv) introduced DualPipe, later open-sourced in the DualPipe repository. It targets Mixture-of-Experts models trained with expert parallelism across nodes, where the all-to-all communication that dispatches tokens to experts and combines their outputs takes about as long as the computation.
DualPipe overlaps the forward pass of one micro-batch with the backward pass of another. Both passes are divided into attention, all-to-all dispatch, MLP and all-to-all combine (with the backward split as in Zero Bubble). These parts are rearranged so that the communication of each pass is hidden behind the computation of the other, with a manually tuned split of the streaming multiprocessors between communication and computation.
To have enough of these pairs, DualPipe uses a bidirectional pipeline: half of the micro-batches enter at the first GPU, and the other half at the last GPU, in the opposite direction. GPU \(i\) therefore holds stages \(i\) and \(p-1-i\), so the pipeline holds two copies of the parameters, whose gradients are summed across GPUs \(i\) and \(p-1-i\) before the optimizer step. DeepSeek notes that, with their large expert-parallel size, this duplication does not increase memory significantly.
The bubble is given by \((p/2-1)(F\&B+B-3W)\), where \(F\&B\) is the time of an overlapped forward-backward pair, and each GPU stores the activations of \(p+1\) micro-batches, whatever their number. Compared to Chimera (ETH Zurich, 2021, arXiv), an earlier bidirectional schedule, DualPipe only requires even numbers of stages and micro-batches, not the micro-batches to be a multiple of the stages.
Note that the diagrams model only computation, so DualPipe’s bubble looks comparable to ZB-H1’s. Its main advantage is that, within each black-bordered pair, the all-to-all communication of expert parallelism is hidden behind computation. DualPipe on 8 GPUs is illustrated below:
DualPipe on 8 GPUs: micro-batches 1 to 8 (light) enter at GPU 0, and micro-batches 9 to 16 (dark) enter at GPU 7. Each GPU holds two pipeline stages. Cells with a shared black border are the forward and backward passes of two different micro-batches executed together, so that the communication of one overlaps with the computation of the other. Adapted from: DeepSeek-V3 Technical Report
DualPipeV
Sea AI Lab later noticed that DualPipe is made of two mirrored halves, as GPUs \(i\) and \(p-1-i\) hold the same stages and run the same schedule (DualPipe could be better without the Dual (Sea AI Lab, 2025)). Keeping only the first half of the GPUs, with all the stages placed on them in a V shape as in ZB-V, gives the same bubble and activation memory as DualPipe for the same number of stages, on half the GPUs and without duplicated parameters.
As in ZB-V, every micro-batch enters at GPU 0, goes down the GPUs and comes back up to the loss on GPU 0, at the price of twice the point-to-point communication of 1F1B. DeepSeek adopted this cut-in-half schedule as DualPipeV in the DualPipe repository, and recent PyTorch versions provide it as ScheduleDualPipeV. It is illustrated below:
DualPipeV with 8 model chunks on 4 GPUs. Light cells refer to the first chunk on each GPU (going down) and dark cells to the second chunk (going back up). Cells with a shared black border are overlapped forward and backward passes of two different micro-batches. Adapted from: DualPipe
Implementing pipeline parallelism with PyTorch
Before using DeepSpeed, the simplest pipeline is implemented in plain PyTorch. The goal is not performance, but to show how activations and gradients flow between the stages. Consider an MLP of n_layers identical layers of size \(d \times d\). On each of the \(p\) GPUs, the model holds only its stage, i.e. \(1/p\) of the layers, in self.layers, with the rank of the GPU in self.rank and \(p\) in self.p.
In the forward pass, each GPU receives the activations of the previous GPU, runs its layers, and sends its output to the next GPU. The first GPU reads the input of the model instead, and the last GPU returns its output for the loss. The received activations are marked with requires_grad_(), so that autograd computes their gradient.
The backward pass runs the same steps in the opposite direction. Each GPU receives the gradient of its output from the next GPU, backpropagates it through its layers, and sends the gradient of its input to the previous GPU. The last GPU starts from the loss instead:
class MLP(nn.Module):
# ...
def forward(self, x):
if self.rank > 0:
x = torch.empty_like(x) # activations have the shape of the input
dist.recv(x, src=self.rank - 1)
x.requires_grad_()
self.inp, self.out = x, self.layers(x) # kept for the backward pass
if self.rank < self.p - 1:
dist.send(self.out.detach(), dst=self.rank + 1)
return self.out
def backward(self, loss=None):
if self.rank == self.p - 1:
loss.backward()
else:
grad_out = torch.empty_like(self.out)
dist.recv(grad_out, src=self.rank + 1)
self.out.backward(grad_out)
if self.rank > 0:
dist.send(self.inp.grad, dst=self.rank - 1)
A training step runs the \(m\) micro-batches one at a time, with the non-overlapped schedule of the first picture. Every GPU calls model(x_mb) and then model.backward(), to which the last GPU passes the loss divided by \(m\), as in gradient accumulation. The gradients of the micro-batches are added, and optimizer.step() updates the layers of each GPU once per mini-batch.
And that is all: the gradients are the same as those of the whole MLP on a single GPU. However, only one GPU computes at a time. 1F1B and GPipe call forward and backward in another order, so they keep the inputs and outputs of several micro-batches. 1F1B also needs non-blocking sends and receives, as two neighboring GPUs then send to each other at the same time.
Implementing pipeline parallelism with DeepSpeed
DeepSpeed implements the 1F1B schedule of PipeDream-Flush, and its schedules can be extended to other algorithms. Here, pipeline parallelism is added to the GPTlite model of the data parallelism post.
DeepSpeed splits into stages any sequence of layers, given as an nn.Sequential container or a list. So GPTlite gets a method to_layers() that returns its layers in the order of its forward pass. As self.blocks is an nn.Sequential, *self.blocks unpacks it into individual blocks, which can be placed on different stages:
## gptlite.py
class GPTlite(nn.Module):
# ...
def to_layers(self):
layers = [
lambda idx:
self.token_embedding_table(idx) +
self.position_embedding_table(torch.arange(idx.shape[1]).to(idx.device)),
*self.blocks,
self.ln,
self.lm_head,
]
return layers
The output of the last layer has shape B,S,C, but CrossEntropyLoss expects the classes in the second dimension. Adding lambda logits: torch.swapaxes(logits, 1, 2) at the end of to_layers() gives the shape B,C,S, but the backward pass may then fail with RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn (DeepSpeed bugs 4274 and 4479), unless outputs = outputs.requires_grad_(True) is added. A cleaner approach is a loss function that reshapes the logits and the labels:
## train.py
class CrossEntropyLoss_FlatView(torch.nn.Module):
def forward(self, logits, labels):
B, S, C = logits.shape
logits = logits.view(B*S, C)
labels = labels.view(-1)
return torch.nn.functional.cross_entropy(logits, labels)
def main_deepspeed(n_epochs=100, random_seed=42):
# ...
criterion = CrossEntropyLoss_FlatView() # initialize loss function
Then, the model is wrapped in a PipelineModule, which is passed to deepspeed.initialize():
## gptlite.py
def get_model(criterion, vocab_size, pipeline_num_stages=0):
# ...
if pipeline_num_stages:
deepspeed.runtime.utils.set_random_seed(random_seed)
pipe_kwargs={
'num_stages': pipeline_num_stages,
'loss_fn': criterion,
}
model = gptlite.GPTlite(vocab_size).to(device_str)
model = deepspeed.pipe.PipelineModule(layers=model.to_layers(), **pipe_kwargs)
else:
# ... as before: model = gptlite.GPTlite(vocab_size)
Finally, the training loop calls engine.train_batch(), which runs the forward passes, backward passes and optimizer step of a whole mini-batch. The number of micro-batches \(m\), which sets the size of the bubble, comes from the field train_micro_batch_size_per_gpu (default: train_batch_size) or gradient_accumulation_steps (default: 1) of the DeepSpeed config file, and is returned by engine.gradient_accumulation_steps():
## train.py
def main_deepspeed(n_epochs=100, random_seed=42):
# ...
for epoch in range(n_epochs):
if pipeline_num_stages:
step_count = len(train_dataset) // engine.gradient_accumulation_steps()
for step in range(step_count):
loss = engine.train_batch()
else:
# ... forward, backward, and update step as before
By default, all mini-batches must have the same shape, as each stage receives the shape of its input only once, with the first micro-batch, and allocates its receive buffers with it. Otherwise, engine.reset_activation_shape() at the start of every mini-batch sends the shape of its first micro-batch again, at the cost of an extra communication step. Up to DeepSpeed 0.14, the micro-batches of a mini-batch cannot have different shapes, and the only workaround is to trim or pad them to the same shape beforehand. Since version 0.15, the argument dynamic_shape=True of PipelineModule sends the shape of every micro-batch instead, at some cost in performance.
Increasing compute and memory efficiency with LayerSpec (optional)
The pipeline above is neither memory efficient nor scalable, as each GPU keeps the whole model in memory. DeepSpeed’s LayerSpec delays the construction of each layer until the layers are partitioned across GPUs, so that each GPU allocates only the layers of its stage. The new class GPTlitePipeSpec inherits from PipelineModule, and its __init__ lists the layers in the order of the forward pass of GPTlite.
LayerSpec only takes subclasses of nn.Module, so the sum of the token and position embeddings is wrapped in the module EmbeddingsSum. The loss function is still CrossEntropyLoss_FlatView:
## gptlite.py
from deepspeed.pipe import PipelineModule, LayerSpec
class GPTlitePipeSpec(PipelineModule):
class EmbeddingsSum(nn.Module):
"""Converts tok_emb + pos_emb into an nn.Module. Required for LayerSpec."""
def __init__(self, vocab_size):
super().__init__()
self.token_embedding_table = nn.Embedding(vocab_size, n_embd)
self.position_embedding_table = nn.Embedding(block_size, n_embd)
def forward(self, idx):
B, S = idx.shape
tok_emb = self.token_embedding_table(idx)
pos_emb = self.position_embedding_table(torch.arange(S).to(idx.device))
return tok_emb + pos_emb
def __init__(self, vocab_size, pipe_kwargs):
self.specs = \
[ LayerSpec(GPTlitePipeSpec.EmbeddingsSum, vocab_size) ] + \
[ LayerSpec(Block, n_embd, n_head) for _ in range(n_layer)] + \
[ LayerSpec(nn.LayerNorm, n_embd),
LayerSpec(nn.Linear, n_embd, vocab_size, bias=False) ]
super().__init__(layers=self.specs, **pipe_kwargs)
The function get_model() then returns this class when pipeline_spec_layers is set:
## gptlite.py
def get_model(criterion, vocab_size, pipeline_num_stages=0, pipeline_spec_layers=False):
if pipeline_num_stages:
if pipeline_spec_layers:
model = GPTlitePipeSpec(vocab_size, pipe_kwargs=pipe_kwargs)
else:
# ... GPTlite model as before
In heterogeneous models, load balancing matters, as the slowest stage sets the pace of the whole pipeline. The stages can be balanced by runtime, memory or parameter count. Here, the load balancing method for pipeline modules is not tuned: the default partition_method=parameters balances the number of parameters, so stages may have different numbers of layers.
Activation checkpointing
To use activation checkpointing, PipelineModule only needs the interval between checkpoints, in layers, in its argument activation_checkpoint_interval:
#gptlite.py
def get_model(criterion, vocab_size, pipeline_num_stages=0, \
pipeline_spec_layers=False, activation_checkpoint_interval=0):
if pipeline_num_stages:
pipe_kwargs={ # ...
'activation_checkpoint_interval': args.activation_checkpoint_interval,
}
# ....
A checkpoint on another GPU than the layers it recomputes would need extra communication. DeepSpeed avoids it by starting the checkpoint intervals at the first layer of each stage, so that every stage starts with a checkpoint, as in GPipe.
Results
Since pipeline parallelism is not compatible with ZeRO stages 2 and 3, the config ds_config.json was changed to run ZeRO stage 1. The runs used 1, 2, 4 or 8 pipeline stages, and the memory-efficient LayerSpec implementation of GPTlite. Activation checkpointing was not used, due to an open bug 4274.
As an example, with 4 stages on 8 GPUs, the partitioning by parameter count gives:
RANK=0 STAGE=0 LAYERS=4 [0, 4) STAGE_PARAMS=21256704 (21.257M)
RANK=2 STAGE=1 LAYERS=3 [4, 7) STAGE_PARAMS=21256704 (21.257M)
RANK=4 STAGE=2 LAYERS=3 [7, 10) STAGE_PARAMS=21256704 (21.257M)
RANK=6 STAGE=3 LAYERS=6 [10, 16) STAGE_PARAMS=21308160 (21.308M)
Memory efficiency of LayerSpec. With the highest stage count (8), the LayerSpec implementation used about 40% less memory than the PipelineModule built from the whole model. This was observed for both GPTlite and the deep benchmark model (2048 layers of 256 neurons).
This code is available in the distributed-ml repo.