Skip to content

Scaling LLM Inference: DP, PP, and TP with vLLM

You sized the model and it doesn't fit on one GPU — or it fits but can't keep up with traffic. Both problems are solved by spreading the model across cards, but with different knobs. Pick the wrong one and you'll waste GPUs, melt your latency, or both. This post is the decision tree.

There are three ways to use more than one GPU, and they answer three different questions:

  • Tensor Parallelism (TP) — the model is too big for one card. Split every layer across GPUs.
  • Pipeline Parallelism (PP) — the model is too big for one node. Split the layers into stages across nodes.
  • Data Parallelism (DP) — the model fits, but you need more throughput. Run independent replicas.

They're not competitors. Real deployments stack them. But you have to know what each one costs before you reach for it.

The three axes at a glance

Splits what? Communication Helps with Hurts
TP Each layer's weights, across GPUs AllReduce every layer Fitting a big model; lower latency Needs fast interconnect (NVLink)
PP Layers into sequential stages Point-to-point between stages Spanning nodes; fitting huge models Pipeline bubble; latency
DP Nothing — full replica per group None during inference1 Throughput, linearly Each replica needs the whole model

Keep one sentence in your head: TP and PP make a model fit; DP makes it go faster.

Tensor Parallelism: shard the layer

TP is the workhorse for single-node multi-GPU. It takes the big matrices inside each layer — the attention projections and the MLP — and splits them column-wise/row-wise across GPUs. Every GPU does a slice of the math for the same token, then they sync.

# One 70B model spread across 4 GPUs in a node
vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 4

Every GPU holds a vertical slice of every layer, works on the same token, and the slices get glued back together at each sync point:

                 one request, one token
                          │
        ┌─────────────────┼─────────────────┐
        ▼                 ▼                 ▼
   ┌─────────┐       ┌─────────┐       ┌─────────┐
   │ GPU 0   │       │ GPU 1   │       │ GPU 2   │   ... GPU 3
   │ heads   │       │ heads   │       │ heads   │
   │ 0–15    │       │ 16–31   │       │ 32–47   │
   │ MLP ¼   │       │ MLP ¼   │       │ MLP ¼   │
   └────┬────┘       └────┬────┘       └────┬────┘
        └────────► AllReduce ◄──────────────┘   ← every layer, ×2
                          │
                  full token state → next layer

The catch is in that "then they sync." TP fires an AllReduce after the attention block and after the MLP — twice per layer, every layer.2 An 80-layer model does ~160 collective syncs per forward pass. That traffic has to cross the GPU interconnect.

  • On NVLink (H100/B200 nodes): ~900 GB/s per link. The syncs are cheap. TP scales beautifully to 8 GPUs.
  • Over PCIe or Ethernet: the AllReduce dominates and your tokens/sec collapses. Never run TP across nodes unless they're joined by NVLink/InfiniBand.

Two hard rules:

  • tensor-parallel-size must divide the attention head count. A model with 64 heads takes TP of 2, 4, 8, 16 — not 6.
  • Keep TP inside one node. TP=8 on an 8-GPU box, good. TP=16 across two boxes over Ethernet, pain.

When to use it: the model won't fit on one card, or it fits but you want lower per-request latency. TP splits the compute for a single request, so it's the only one of the three that makes an individual request faster.

Pipeline Parallelism: stage the layers

PP cuts the model the other way. Instead of slicing every layer, it assigns whole blocks of layers to each GPU. Layers 1–20 on GPU 0, 21–40 on GPU 1, and so on. A request flows through them like an assembly line.

# 16 GPUs across 2 nodes (8 each): TP within a node, PP across nodes
vllm serve meta-llama/Llama-3.1-405B-Instruct \
  --tensor-parallel-size 8 \
  --pipeline-parallel-size 2

Each GPU holds a horizontal slice — whole layers, the full width — and passes the activation downstream like an assembly line:

 request ─► ┌─────────┐ ─► ┌─────────┐ ─► ┌─────────┐ ─► ┌─────────┐ ─► token
            │ GPU 0   │    │ GPU 1   │    │ GPU 2   │    │ GPU 3   │
            │ layers  │    │ layers  │    │ layers  │    │ layers  │
            │ 1–20    │    │ 21–40   │    │ 41–60   │    │ 61–80   │
            └─────────┘    └─────────┘    └─────────┘    └─────────┘
              stage 0        stage 1        stage 2        stage 3
                 └──── point-to-point handoff (small tensor) ────┘

  Bubble: with 1 request in flight, only 1 stage is busy; the other 3 idle.
  vLLM fills the pipe with concurrent requests so every stage stays hot.

The communication is the opposite of TP: instead of a fat AllReduce every layer, PP just hands the activation tensor from one stage to the next — a single point-to-point send at each stage boundary.3 That's tiny, which is exactly why PP tolerates slow links and TP doesn't.

The cost shows up as the pipeline bubble. While stage 0 works on the first token, stages 1–3 sit idle waiting for it. vLLM hides this by keeping the pipeline full of concurrent requests — but if your traffic is thin, those downstream GPUs are warming the room for free.

The canonical multi-node recipe is one line worth memorizing:

TP = GPUs per node. PP = number of nodes.4

You keep the chatty AllReduce traffic on the fast NVLink inside each node, and you only push the cheap point-to-point traffic between nodes. 405B on two 8×H100 boxes → --tensor-parallel-size 8 --pipeline-parallel-size 2.

When to use it: the model is too big even for a full node, or you're scaling across nodes joined by something slower than NVLink.

Going multi-node: how vLLM finds your second machine

That --pipeline-parallel-size 2 only works if vLLM can see both machines. It can't do that on its own — vLLM doesn't SSH around or probe the network for GPUs. You join the machines into a single Ray cluster first, and vLLM consumes whatever GPU pool Ray reports.5

   VM-1 (head)                         VM-2 (worker)
 ┌──────────────┐                    ┌──────────────┐
 │ ray start    │                    │ ray start    │
 │   --head     │ ◄───── joins ──────│  --address=  │
 │  GPUs 0–7    │   one Ray cluster  │  GPUs 0–7    │
 └──────┬───────┘                    └──────────────┘
        │
   vllm serve  ← run ONCE, on the head only
        │
   Ray places workers on VM-2 automatically; vLLM sees 16 GPUs

Step 1 — start the cluster. Pick one VM as the head, the other as a worker.

# On VM-1 (head). Prints the address to join with.
ray start --head --port=6379

# On VM-2 (worker). Point it at the head's IP.
ray start --address='10.0.0.1:6379'

Step 2 — confirm Ray sees both. This total is exactly what vLLM will use.

ray status     # expect 2 nodes, 16 GPUs total

Step 3 — launch vLLM once, on the head node.

vllm serve meta-llama/Llama-3.1-405B-Instruct \
  --tensor-parallel-size 8 \
  --pipeline-parallel-size 2 \
  --distributed-executor-backend ray

TP × PP must equal the GPU count Ray reported (8 × 2 = 16). No Ray cluster → vLLM only ever sees the local VM's GPUs.

What has to be true for this to actually work:

  • Identical vLLM, CUDA, and model versions on both VMs. Mismatches surface as cryptic NCCL crashes on the workers.
  • The model is reachable from both VMs — same HF cache path, a shared NFS mount, or pre-downloaded on each. Every worker loads its own shard.
  • The inter-VM link is fast and open. NCCL traffic flows VM↔VM; InfiniBand/RDMA is ideal. This is why you keep TP inside a VM and PP across them — PP's tiny handoffs tolerate the slow link, TP's per-layer AllReduce would choke on it.
  • Firewall allows the Ray + NCCL ports between VMs (6379 for Ray, plus the NCCL/GLOO ranges).

If NCCL grabs the wrong network interface, pin it:

export NCCL_SOCKET_IFNAME=eth0   # the NIC the VMs talk over
export GLOO_SOCKET_IFNAME=eth0

vLLM ships a run_cluster.sh Docker helper that wraps the ray start dance if you'd rather not run it by hand.4 The mental model to keep: Ray is what "knows" about both machines; vLLM just consumes the GPU pool Ray exposes.

Data Parallelism: replicate and load-balance

DP is the simplest idea and the one people reach for last, wrongly. It does nothing to make a model fit — every replica loads the entire model. What it buys you is throughput that scales close to linearly with the number of replicas.

# 4 full replicas, each on 2 GPUs (TP=2) → 8 GPUs total
vllm serve mistralai/Mistral-Small-3.2-24B-Instruct-2506 \
  --data-parallel-size 4 \
  --tensor-parallel-size 2

Each replica is a complete, self-contained copy of the model; the only shared piece is the load balancer in front:

              incoming requests
                     │
              ┌──────┴───────┐
              │ load balancer│
              └──┬───┬───┬───┬┘
                 │   │   │   │
        ┌────────┘   │   │   └────────┐
        ▼            ▼   ▼            ▼
   ┌─────────┐  ┌─────────┐      ┌─────────┐
   │replica 0│  │replica 1│ ...  │replica 3│
   │ FULL    │  │ FULL    │      │ FULL    │
   │ model   │  │ model   │      │ model   │
   └─────────┘  └─────────┘      └─────────┘
       no comms between replicas — fully independent

For dense models, the replicas are independent during inference — zero cross-replica communication.1 A built-in load balancer fans incoming requests across them. Double the replicas, roughly double the tokens/sec.

When to use it: the model already fits (on one GPU, or on a TP group), and your bottleneck is concurrent request volume, not model size. This is the throughput knob.

The decision tree

Walk it top to bottom:

  1. Does the model fit on one GPU? → Yes: just need more throughput? DP. Done. → No: keep going.
  2. Does it fit on one node (with TP across the node's GPUs)? → Yes: TP up to the GPU count. Add DP on top if one TP group can't handle the load.
  3. Too big for a node? → TP inside each node + PP across nodes. Layer DP on top only if you have spare nodes and need more throughput.
Symptom Reach for
OOM loading weights, single card TP
OOM even on a full 8-GPU node TP + PP
Fits fine, but queue depth is climbing DP
Need the lowest possible single-request latency TP (max it within the node)
Slow interconnect between your machines PP across, never TP

MoE changes the math: Expert Parallelism

Mixture-of-Experts models (DeepSeek-V3, Qwen3-235B, Llama-4) have a twist. Most of the parameters live in experts, and each token only touches a couple of them. Sharding those experts the dense-TP way wastes memory and bandwidth, so vLLM adds Expert Parallelism (EP).

EP isn't a standalone mode — it's a flag that changes how the MoE layers shard once you already have TP or DP going.6 The two patterns:

  • TP + EP (--tensor-parallel-size 8 --enable-expert-parallel) — experts split across GPUs, syncs via AllReduce. KV cache gets duplicated on every rank, which is brutal for MLA models like DeepSeek. Best at low-to-moderate concurrency, where latency wins.
  • DP attention + EP (--data-parallel-size 8 --enable-expert-parallel) — attention runs data-parallel so each GPU holds only 1/N of the KV cache, while experts route tokens across GPUs via AllToAll. Best at high concurrency, and effectively required for MLA models to avoid the KV duplication tax.7

The crossover sits around 256–512 concurrent requests: below it, TP+EP usually wins; above it, DP+EP pulls ahead.7 One non-obvious gotcha — if a model is ultra-sparse (each token hits well under 1% of experts), the AllToAll shuffle can cost more than it saves; run plain TP without EP.7

# DeepSeek-style MLA model, high-concurrency serving
vllm serve deepseek-ai/DeepSeek-V3 \
  --data-parallel-size 8 \
  --enable-expert-parallel \
  --max-model-len 32768

Worked examples

70B on a single 4×H100 node, chat workload. Model is ~140 GB at FP16, won't fit on one 80 GB card. TP across the node, nothing fancy. --tensor-parallel-size 4

8B model, hammered by traffic. Fits on one card with room to spare. Pure throughput play — replicate. --data-parallel-size 4 (4 cards, 4 replicas)

24B, needs both fit and throughput. Two cards per replica for headroom, four replicas for volume. --data-parallel-size 4 --tensor-parallel-size 2 (8 GPUs)

405B across two 8-GPU nodes. Too big for one node. Fast NVLink stays inside each node via TP; the slow inter-node link only carries stage handoffs via PP. --tensor-parallel-size 8 --pipeline-parallel-size 2

Gotchas that bite

  • TP across PCIe/Ethernet tanks throughput. The per-layer AllReduce is the whole reason. If nvidia-smi topo -m shows your GPUs talking over PCIe, not NVLink, cap TP at the NVLink island.
  • PP adds latency even when it "works." A request now traverses every stage serially. Great for batch throughput, worse for a single interactive request than the same GPUs in TP.
  • DP multiplies your VRAM bill. N replicas = N full copies of the weights and N separate KV-cache pools. It scales throughput, never memory.
  • tensor-parallel-size must divide the head count, not just be a power of two. Check num_attention_heads in config.json first.
  • Total GPUs = DP × TP × PP. vLLM will refuse to start if that product doesn't match the GPUs it can see. Do the multiplication before you launch.
  • EP does nothing unless TP × DP > 1. On a single GPU the flag is a no-op.

Summary

The whole decision on one card:

  • Fits on one GPU, need speed? → DP. Replicate, load-balance, scale throughput linearly.
  • Too big for one GPU? → TP, capped at the GPU count and the NVLink island.
  • Too big for one node? → TP inside nodes + PP across nodes. Memorize: TP = GPUs/node, PP = nodes.
  • MoE model? → add --enable-expert-parallel: TP+EP for low concurrency, DP+EP for high concurrency / MLA models.
  • Sanity check: total GPUs = DP × TP × PP, and tensor-parallel-size divides the attention head count.
  • Golden rule: TP and PP make a model fit; DP makes it go faster. Don't confuse the two.

Size the model first — that's the VRAM field guide — then pick the parallelism that matches why one GPU isn't enough. The flag is one word; choosing it for the wrong reason is a rack of idle GPUs.

Questions or discussion? Connect on LinkedIn, X or reach out via email.

  1. vLLM, Data Parallel Deployment — for dense models, data-parallel replicas run independently with a built-in request load balancer; cross-replica communication only appears in the MoE "DP attention" case. ↩↩

  2. vLLM, Parallelism and Scaling — tensor parallelism shards layers across GPUs and synchronizes with an AllReduce after the attention and MLP blocks each layer; recommended within a single node over NVLink. ↩

  3. vLLM, Parallelism and Scaling — pipeline parallelism splits the model into sequential layer stages connected by point-to-point transfers, tolerating slower interconnects but introducing a pipeline bubble that vLLM hides by batching concurrent requests. ↩

  4. vLLM, Distributed Inference and Serving — common practice is to set tensor-parallel-size to the number of GPUs per node and pipeline-parallel-size to the number of nodes (e.g. 16 GPUs over 2 nodes → TP 8, PP 2); ships a run_cluster.sh Docker helper to launch the Ray cluster. ↩↩

  5. vLLM, Distributed Inference and Serving — multi-node serving runs on a Ray cluster: ray start --head on one node, ray start --address=... on the others, then vllm serve on the head node, which discovers all GPUs via Ray. ↩

  6. vLLM, Expert Parallel Deployment — --enable-expert-parallel switches MoE layers to expert parallelism; it only changes behavior when TP × DP > 1, otherwise MoE layers fall back to tensor parallelism. ↩

  7. AMD ROCm Blogs, The vLLM MoE Playbook: TP, DP, PP and Expert Parallelism — TP+EP wins at low concurrency (≤128 reqs) and DP+EP at high concurrency (≥512), with a crossover near 256–512 requests; DP attention avoids KV-cache duplication for MLA models, and ultra-sparse experts (<1% activation) can be faster without EP. ↩↩↩

Discussion

Have thoughts on this post? Share them below — questions, corrections, or your own experience are all welcome.