Distributed Inference (TP/PP)

1. Definition: The "Sharding Art" When One Card Can't Hold the Model

Distributed inference means splitting a complete model across multiple GPUs that cooperate on inference. The reason is simple: a single card's memory can't hold the model. Llama-3-70B in FP16 needs ~140 GB of GPU memory — more than one A100 80GB can offer; GPT-4-class models are inferred across thousands of GPUs. Anything beyond a single card must be sharded.

The essence of sharding: "cut" the model along some dimension so that each card holds only part of the weights / KV cache, while externally it still behaves like one model. Three ways to cut:

Model (stacked layers, each with attention / FFN)
   │
   ├── Split 1: by layer  → pipeline parallelism (PP)
   │     Layer 0-15 on GPU 0
   │     Layer 16-31 on GPU 1
   │     ...
   │
   ├── Split 2: by weight matrix  → tensor parallelism (TP)
   │     each attention head on a different GPU
   │     each GEMM split by row / column
   │
   └── Split 3: by MoE expert  → expert parallelism (EP)
         Expert 0-3 on GPU 0
         Expert 4-7 on GPU 1

Split	What is split	When communication happens	Communication primitive	Difficulty
TP	Weight matrices	After each layer's attention / FFN	AllReduce	Medium
PP	Layers	Between micro-batches	P2P Send/Recv	High
EP	Experts	At token routing	All-to-All	Extreme
SP	Sequence	During attention	AllGather / AllReduce	High

We dissect each below; see GPU Architecture and Optimization, Model Serving and Orchestration, and Batching and Request Scheduling.

2. Tensor Parallelism (TP)

Core Idea: Split Weight Matrices Along a Dimension

TP was proposed in Megatron-LM (NVIDIA, 2019). The idea is to cut a weight matrix into N pieces, with each of the N cards holding one piece.

Splitting Attention (by Head)

Llama-3-8B has 32 attention heads; with 4-way TP, each card holds 8 heads:

Original:    Q,K,V  [seq_len, 32 * d_head]
                 │
                 ▼
                 attention computation (one forward pass)
                 │
                 ▼
                 output [seq_len, d_model]

TP=4 split: GPU 0 holds heads 0-7
            GPU 1 holds heads 8-15
            GPU 2 holds heads 16-23
            GPU 3 holds heads 24-31

            each card computes attention independently (no communication!)
                 │
                 ▼
            output [seq_len, d_model/N]   ← each card has only 1/N of the output

            AllReduce ← sum the 4 pieces → complete [seq_len, d_model]

Key point: attention heads are independent, so TP-by-head adds zero computation — it only needs one AllReduce at the end of each layer.

Splitting the FFN (by Matrix)

An FFN is two GEMMs: y = W2 * activation(W1 * x). The TP split:

W1 split by row:    W1 = [W1_0; W1_1; W1_2; W1_3]      ← split along the output dimension
W2 split by column: W2 = [W2_0 | W2_1 | W2_2 | W2_3]   ← split along the input dimension

Each card i holds W1_i, W2_i:
   h_i = activation(W1_i @ x)        ← input x is identical on all cards; output h_i has 1/N dimension
   y_i = W2_i @ h_i                  ← output y_i has d_model dimension

   y = sum_i(y_i)                     ← AllReduce

TP performs one AllReduce per layer across attention + FFN, so the larger the TP degree, the more communication overhead grows linearly. Rule of thumb: TP ≤ the number of GPUs in one machine (connected via NVLink); go with PP or EP across machines.

TP in vLLM / TRT-LLM

# Launch Llama-3-70B with TP=2 in vLLM
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Meta-Llama-3-70B-Instruct \
    --tensor-parallel-size 2

# Split at TRT-LLM build time
trtllm-build ... --tp_size 2

TP is usually capped at 8

An 8-GPU machine (e.g., DGX A100/H100) is interconnected with NVLink + NVSwitch, where AllReduce latency is extremely low (~1 μs). Beyond 8 cards you must cross machines over InfiniBand, where AllReduce latency rises to tens of μs and TP gains collapse. Use PP or EP for cross-machine scenarios.

3. Pipeline Parallelism (PP)

Core Idea: Split by Layer, Pipeline Micro-batches

PP cuts the model into N segments by layer, one segment per card:

GPU 0: Layer 0-15  →  GPU 1: Layer 16-31  →  GPU 2: Layer 32-47  →  GPU 3: Layer 48-63

Data flow:  micro-batch_1 → GPU 0 → GPU 1 → GPU 2 → GPU 3 → output_1
            micro-batch_2 → GPU 0 → GPU 1 → GPU 2 → GPU 3 → output_2

Communication is light (only P2P Send/Recv between stages), but there are bubbles — at any moment only one stage is busy:

Time:    0  1  2  3  4  5  6  7  8  9
GPU 0:   B1 .  .  B2 .  .  B3 .  .  .
GPU 1:   .  B1 .  .  B2 .  .  B3 .  .
GPU 2:   .  .  B1 .  .  B2 .  .  B3 .
GPU 3:   .  .  .  B1 .  .  B2 .  .  B3
               ↑              ↑
               bubble         bubble

1F1B Scheduling: Shrinking the Bubble

1F1B (One Forward One Backward) scheduling interleaves forward and backward passes (inference applies a similar idea with micro-batching), shrinking the bubble fraction from ~50% to ~1/N.

1F1B schedule (4 stages, 4 micro-batches):
Time:    0  1  2  3  4  5  6  7
GPU 0:   F1 F2 F3 F4 .. .. .. ..
GPU 1:   .. F1 F2 F3 F4 .. .. ..
GPU 2:   .. .. F1 F2 F3 F4 .. ..
GPU 3:   .. .. .. F1 F2 F3 F4 ..
bubble fraction ~1/N (N = number of micro-batches)

PP is less popular for inference than for training — inference has only the forward pass, so bubbles are harder to eliminate. LLM inference typically uses PP for "very large models beyond 70B + cross-machine deployment," combined with TP as hybrid parallelism:

2D parallelism: TP × PP
   TP=4 (within the node) + PP=2 (across nodes) = 8 GPUs

GPU 0 (TP rank 0, PP stage 0)  ── GPU 4 (TP rank 0, PP stage 1)
GPU 1 (TP rank 1, PP stage 0)  ── GPU 5 (TP rank 1, PP stage 1)
GPU 2 (TP rank 2, PP stage 0)  ── GPU 6 (TP rank 2, PP stage 1)
GPU 3 (TP rank 3, PP stage 0)  ── GPU 7 (TP rank 3, PP stage 1)
       NVLink within node A           NVLink within node B
                ↕ InfiniBand interconnect

4. Expert Parallelism (EP): Dedicated to MoE Models

MoE (Mixture of Experts) models such as Mixtral-8x22B and DeepSeek-V3 have N experts, and each token is routed to 2–8 experts. EP distributes the experts across cards:

Mixtral-8x7B (8 experts) + 4-way EP:
   GPU 0: Expert 0, 1
   GPU 1: Expert 2, 3
   GPU 2: Expert 4, 5
   GPU 3: Expert 6, 7

Each token is routed → All-to-All communication → tokens go to the expert's GPU → computed and shipped back

EP communication is All-to-All (every card sends tokens to every other card), and the volume exceeds TP's AllReduce. See Classic Papers in Depth and Frontier Advances.

DeepSeek-V3's MTP + EP + TP hybrid parallelism is the current best practice for MoE inference:

DeepSeek-V3 inference architecture (671B total parameters, 37B activated):
   TP=8 (within the node, for the attention part)
   EP=64 (across nodes, for the expert part)
   Total GPUs = 8 × 64 = 512 cards

5. Sequence Parallelism (SP)

SP is used for long-context scenarios (128K+ tokens), splitting one sequence along the sequence dimension across cards, with each card computing a segment.

• DeepSpeed Ulysses: splits along the head dimension; attention reorganizes with All-to-All
• Ring Attention: ring communication with rotating KV blocks

SP is mainly used in long-context LLM training; inference applications are rare — long-context inference more often uses chunked prefill (vLLM) or inference variants of RingAttention.

6. Communication Costs: Hardware Interconnect Decides Everything

The "bottleneck" of distributed inference is almost always communication:

Interconnect	One-way latency	One-way bandwidth	Applicability
NVLink 4 (H100)	~1 μs	900 GB/s	Single-node TP, optimal
NVLink 3 (A100)	~1.5 μs	600 GB/s	Single-node TP
NVLink NVSwitch	~1 μs	900 GB/s	Single-node all-to-all TP
PCIe Gen5	~5 μs	64 GB/s	Constrained single-node TP, poor cross-GPU communication
InfiniBand HDR	~0.5 μs	200 Gb/s (25 GB/s)	Cross-node PP / EP
InfiniBand NDR	~0.5 μs	400 Gb/s (50 GB/s)	Cross-node PP / EP, newer clusters
RoCE v2	~1 μs	100-400 Gb/s	Cross-node alternative

Never do TP across machines

NVLink bandwidth is 10–35× that of InfiniBand, and TP's AllReduce would be completely choked by InfiniBand communication. TP strictly within a node, PP/EP across nodes is the iron rule of distributed inference.

7. Mainstream Implementations

Framework	TP	PP	EP	Notes
vLLM	✅	✅ (v0.6+)	✅ (v0.7+)	First choice in production, broadest support
TensorRT-LLM	✅	✅	✅	Extreme performance on NVIDIA H100
DeepSpeed-MII	✅	✅	✅	Microsoft stack; ZeRO-Inference supports beyond-memory models
SGLang	✅	✅	✅	Strong for complex applications
TGI	✅	❌	❌	Official HF offering, TP only
Megatron-LM	✅	✅	✅	Training-first, inference-capable
RayLLM / vLLM on Ray	-	-	-	Multi-node scheduling layer

vLLM Multi-GPU Deployment Example

# Single-node 4-GPU TP
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Meta-Llama-3-70B-Instruct \
    --tensor-parallel-size 4

# Cross-node PP (vLLM 0.6+)
# Node 0:
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Meta-Llama-3-70B-Instruct \
    --pipeline-parallel-size 2 \
    --tensor-parallel-size 4 \
    --node-rank 0 \
    --nnodes 2

# Node 1:
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Meta-Llama-3-70B-Instruct \
    --pipeline-parallel-size 2 \
    --tensor-parallel-size 4 \
    --node-rank 1 \
    --nnodes 2

DeepSpeed-MII: Beyond-Memory Inference

DeepSpeed-MII's ZeRO-Inference can run models that exceed a single machine's GPU memory — by sharding weights to CPU + NVMe and loading them onto the GPU on demand. The cost is high latency, but it runs:

import mii
mii.serve(
    "meta-llama/Meta-Llama-3-70B",
    deployment_name="llama3-70b",
    tensor_parallel=4,
    enable_zero=True,                # enable ZeRO-Inference
    max_concurrent_tokens=64,
)

8. Performance Data: Baseline Reference

Here is a set of distributed-inference baselines for Llama-3-70B on A100 80GB (see Inference Benchmarking in Practice):

Deployment	Precision	Concurrency	Throughput (tokens/s)	First-token latency
Single card 80GB FP16	-	-	OOM	-
Single card 80GB INT4 AWQ	1	60	50 ms
Single card 80GB INT4 AWQ	32	1200	75 ms
2-card TP=2 FP16	1	100	30 ms
2-card TP=2 FP16	32	2200	35 ms
4-card TP=4 FP16	32	4000	30 ms
4-card TP=4 FP16	128	7000	40 ms
8-card TP=8 FP16	128	8500	30 ms
2 nodes, 8 cards, TP=8 PP=2 FP16	128	6500	60 ms	← cross-node PP penalty

Observations:

• TP=4 reaches 4000 tokens/s; raising concurrency to 128 gets 7000 — close to the NVLink communication ceiling;
• Cross-node PP=2 drops throughput to 6500, ~25% below single-node 8-card TP=8 — the cost of PP bubbles and cross-machine communication.

9. Limitations and Boundaries

1. TP capped at 8 (single node): cross-machine TP gets choked by communication; you must use PP or EP.
2. PP bubbles are hard to eliminate: inference bubbles are harder to remove than training bubbles (inference has only the forward pass); filling the pipeline needs high concurrency.
3. EP complexity is high: All-to-All communication + load balancing is hard; uneven expert distribution leaves some GPUs idle.
4. Complex scheduling: cross-machine coordination, failure recovery, and scale-up/down are several times more complex than single-machine inference.
5. Cost: an 8-GPU H100 node costs ~$300,000, and a 128-GPU cluster ~$5 million — few businesses can justify that.
6. Little gain for small requests: for a single user at batch=1, distributed inference is barely faster — communication overhead cancels the sharding gains. The sweet spot of distributed inference is high-concurrency production serving.
7. Beyond-memory fallback: when memory truly runs out, ZeRO-Inference can park weights on CPU/NVMe, but latency goes to seconds — suitable only for "occasionally running long contexts," not real-time interaction.

10. Decision Tree: What to Use When

Does the model fit in a single GPU's memory?
   ├─ Yes → single GPU ([vLLM](/case-studies/vllm) / [TensorRT-LLM](/case-studies/tensorrt-llm) / [llama.cpp](/case-studies/llama-cpp))
   └─ No → sharding required
            │
            ├─ Does the model fit in a single node's 8-GPU memory?
            │   ├─ Yes → single-node TP (vLLM --tensor-parallel-size 2/4/8)
            │   └─ No → cross-node
            │
            ├─ Is it a MoE model?
            │   ├─ Yes → EP + TP hybrid
            │   └─ No → TP=8 within the node + PP across nodes
            │
            ├─ Long context (128K+)?
            │   └─ consider SP (Ring Attention / DeepSpeed Ulysses)
            │
            └─ Really out of memory → ZeRO-Inference (CPU offload, trading latency for memory)

11. Where to Go Next

• Concept pages: GPU Architecture and Optimization, The GPU Memory Hierarchy and the Bandwidth Wall, Latency, Throughput, and Concurrency, Model Serving and Orchestration, Batching and Request Scheduling
• Case-study pages: vLLM, TensorRT-LLM, Triton Inference Server, Speculative Decoding and Medusa/EAGLE
• Papers: Classic Papers in Depth (Megatron-LM / DeepSpeed / GPipe deep dives), Reading Paths, Frontier Advances
• Practice pages: Inference Engine Comparison, Tuning and Performance Optimization, Inference Benchmarking in Practice, Portfolio Projects, Deployment Design Principles, Common Pitfalls and Anti-Patterns
• Resource pages: Hardware Primer, Benchmark Data & Tool Profiles, Curated Resources

References

• Shoeybi et al. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019) — the TP classic
• Huang et al. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism (NeurIPS 2019) — the PP classic
• Narayanan et al. Efficient Large-Scale Language Model Training on GPU Clusters using Megatron-LM (SC 2021) — 1F1B scheduling
• Fedus et al. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (2021) — MoE + EP
• DeepSeek-AI. DeepSeek-V3 Technical Report (2024.12) — MTP + EP engineering practice
• Microsoft. DeepSpeed — ZeRO-Inference implementation
• NVIDIA. Megatron-Core — TP/PP/EP training and inference stack
• vLLM Distributed Serving documentation — vLLM multi-GPU deployment