Gpu for llm workloads reference

LLM workloads are not "regular workloads that happen to need more RAM." They are memory-bandwidth-bound during decode, compute-bound during prefill, and topology-bound the moment a model can't fit on one GPU. The hardware spec sheet, the VM config, the container runtime, the Kubernetes device plugin, and the multi-GPU pattern you pick are all the same decision at different layers. Get one wrong and the others stop mattering.

This is a reference for the whole stack — from the HBM3e bandwidth of a Blackwell GPU to the nvidia.com/gpu resource advertised by a DaemonSet to the tensor-parallel group size that decides whether your 70B model serves or stalls. Every number is verified against a primary source, dated 2026-06-16.

Glossary¶

The post is dense. If a term below trips you up, jump back here.

Hardware & memory

• VRAM — video RAM on the GPU. Model weights and KV cache live here. Larger is better.
• HBM, HBM2e, HBM3, HBM3e — High-Bandwidth Memory. Stacked DRAM that sits next to the GPU die, giving 1–8 TB/s bandwidth. Each generation roughly doubles the bandwidth. HBM3e (used on H200, B200) is the current top tier.
• GDDR6, GDDR7 — cheaper, more common graphics memory. Lower bandwidth per GB than HBM but cheaper to make and available in larger capacities. L40S = GDDR6; RTX PRO 6000 = GDDR7.
• FP8 / FP16 / FP4 / BF16 / TF32 — numeric precisions. Lower = less memory, faster math, slightly less accurate. FP8 is the new norm for LLM inference (vs FP16) because it halves memory traffic and doubles tensor-core throughput. FP4 is Blackwell-only.
• TDP — Thermal Design Power, the watts a GPU draws at full load. 350W = air-cooled in a normal rack; 1000W+ = needs liquid cooling.
• NVLink — NVIDIA's proprietary GPU-to-GPU interconnect (900 GB/s on Hopper, 1.8 TB/s on Blackwell). Faster than PCIe but only between NVIDIA GPUs.
• NVSwitch — the chip that ties 8 NVLink GPUs into a fully connected fabric (e.g. inside an HGX H100 baseboard).
• PCIe — the standard slot your GPU plugs into. PCIe Gen5 = 128 GB/s. 8–10× slower than NVLink, which is why data center GPUs use SXM/NVLink instead.
• SXM vs PCIe — SXM is a mezzanine connector with more pins, more NVLink lanes, and higher TDP budget. PCIe is the standard slot.

LLM-specific

• Prefill — the "read the prompt" phase. Compute-bound, parallelizes well.
• Decode — the "generate token by token" phase. Memory-bandwidth-bound, hard to parallelize.
• KV cache — the running memory of attention keys/values. Grows with context length × batch size. Must fit in VRAM alongside weights, or inference falls apart.
• PagedAttention — vLLM's technique for managing KV cache in non-contiguous memory pages, like OS virtual memory. Lets you serve many requests with one model.
• MoE (Mixture of Experts) — an architecture where only some of the model's "experts" run per token. Qwen3-Coder-30B-A3B has 30B total params but only 3B active per token. Saves compute, costs memory.
• Tensor-parallel (TP), Pipeline-parallel (PP), Data-parallel (DP), Expert-parallel (EP) — ways to split one model across many GPUs. §7 covers them in detail.

Virtualization & sharing

• VFIO passthrough — VM-level direct GPU access via IOMMU. Highest performance, no hypervisor in the data path. Different from K8s "passthrough" — see §6.
• IOMMU — Input/Output Memory Management Unit. The hardware feature that makes VFIO passthrough possible (AMD = AMD-Vi, Intel = VT-d).
• MIG (Multi-Instance GPU) — hardware partition of a single GPU into 2–7 fully isolated instances, each with its own SMs and memory. A100/H100/H200/B200/RTX PRO 6000 only. L40S does not support MIG.
• vGPU — licensed NVIDIA product that gives a VM a fraction of a physical GPU with hardware isolation. Costs ~$1K+/GPU/year. Supported on L40S, A100, RTX PRO 6000.
• Time-slicing — software-only GPU sharing. Multiple pods take turns in 1ms slices. No isolation. Free.
• CUDA MPS — Multi-Process Service. One CUDA context shared by multiple processes.
• SR-IOV — Single Root I/O Virtualization. Hardware-isolated virtual functions of a PCIe device. AMD MI300X supports it; NVIDIA does not.
• vGPU (re-stated for cross-link) — see above.

Containers & Kubernetes

• OCI runtime — the low-level component that actually creates a container (e.g. runc, crun, nvidia-container-runtime). Distinct from containerd/Docker, which are higher-level.
• CDI (Container Device Interface) — a CNCF spec for how to expose devices (GPUs, NICs) to containers. The modern path; replaces the legacy NVIDIA_VISIBLE_DEVICES env-var approach.
• Pod — the smallest deployable unit in K8s. Usually one container, sometimes a few tightly-coupled ones.
• DaemonSet — a K8s controller that runs one pod per node. GPU Operator's components all run as DaemonSets.
• kubelet — the K8s agent on each node. Calls the device plugin's Allocate RPC when a pod needs a GPU.
• Helm — a package manager for K8s. The GPU Operator is installed via helm install.
• Operator — a K8s pattern where a controller watches custom resources (CRDs) and reconciles them. GPU Operator, Karpenter, Kubeflow Trainer are all Operators.
• Device plugin — a K8s framework for advertising specialized hardware (GPUs, FPGAs, NICs) as schedulable resources. The NVIDIA device plugin registers nvidia.com/gpu.
• DRA (Dynamic Resource Allocation) — the K8s 1.30+ replacement for device plugins. Uses ResourceClaim objects that describe what a pod needs (e.g. "H100 with ≥80GB"); a DRA driver matches. Beta default-on in K8s 1.36, use 1.36.2+ to avoid a double-allocation bug.
• ResourceClaim / ResourceSlice / DeviceClass — the DRA primitives. ResourceClaim is per-pod, ResourceSlice describes a device pool, DeviceClass is a class of device.
• ComputeDomain — a DRA abstraction for Multi-Node NVLink. The NVIDIA DRA driver uses it to make a 72-GPU NVL72 act as one scheduling unit.
• Karpenter — a K8s node provisioner. Faster than Cluster Autoscaler, the right answer for GPU nodes.
• Multus CNI — a CNI plugin that lets a pod have multiple network interfaces (e.g. one normal, one RDMA fabric).
• SR-IOV CNI — CNI for binding SR-IOV virtual functions into pods.
• DCGM — NVIDIA Data Center GPU Manager. The library that exposes GPU metrics (temperature, util, ECC errors, XID codes).
• Prometheus — the time-series database that scrapes those DCGM metrics.
• NCCL — NVIDIA's collective communication library (all-reduce, all-gather, etc.). What PyTorch uses under the hood for distributed training.
• InfiniBand / RoCE — high-speed network fabrics. InfiniBand = dedicated, lower latency. RoCE = RDMA over Converged Ethernet, cheaper.
• RDMA — Remote Direct Memory Access. Lets a NIC read/write GPU memory directly, bypassing the CPU. Required for multi-node training.

1. What "GPU for LLM" actually means¶

The first-order metrics for sizing LLM infrastructure are not the same as for training a vision model or running a database:

1. VRAM capacity — model weights + KV cache must fit, or you page to host RAM / NVMe, which adds 10–100× latency.
2. HBM bandwidth — drives decode throughput (tokens/sec/user) once weights are loaded. For a 70B model on H100 SXM, decode is 100% memory-bandwidth-bound.
3. Inter-GPU bandwidth (NVLink) — drives tensor-parallel efficiency. Below ~600 GB/s aggregate all-to-all, TP scaling collapses.
4. FP8/FP4 tensor throughput — doubles FP16 throughput on Hopper/Blackwell; critical for serving at scale.
5. Host-to-GPU link (PCIe / NVLink-C2C) — matters for model loading, not for steady-state inference.

The math: a 70B model in FP16 needs ~140 GB just for weights. The H100 NVL ships 94 GB specifically because it needed to fit Llama-2-70B FP16 with a usable KV cache. The H200's 141 GB is the response to "we want one GPU, not two, for inference of 70B-class models."

The bandwidth plateau: H100 SXM = 3.35 TB/s. H200 SXM = 4.8 TB/s. B200 SXM ≈ 8 TB/s per GPU. The per-generation gain in memory bandwidth is real but smaller than the FLOPS gain — which is why FP8 and FP4 matter: you do the same memory work in less time.

2. Physical layer: the silicon¶

2.1 NVIDIA line-up (June 2026)¶

All spec numbers verified against NVIDIA's .md mirror on nvidia.com (2026-06-16).^12202122345

Three things this table tells you:

1. Memory bandwidth is the actual ceiling. A100 = 2.04 TB/s, H100 SXM = 3.35 TB/s, H200 = 4.8 TB/s, B200 ≈ 8 TB/s. The +43% from H100 → H200 was the single biggest change in two years. FP8/FP4 exist because we can't make HBM faster.
2. HBM is not the only memory type. A100, H100, H200, B200, GB200 all use HBM (or HBM2e/HBM3/HBM3e). L40S uses GDDR6 (cheaper, 1/4 the bandwidth per GB) — fine for batch=1 inference of 7B models, but decode throughput on 30B+ models is bandwidth-bound. RTX PRO 6000 uses GDDR7 — a 2026 hybrid: 96 GB capacity close to H200's 141 GB, at 1.6 TB/s bandwidth (between H100 and H200). The trade-off is real: GDDR7 capacity scales cheaper than HBM3e, but you pay in bandwidth.
3. NVLink is the gating factor for multi-GPU. 600 GB/s (A100) is the floor. 900 GB/s (H100/H200) is fine for TP8. 1.8 TB/s (Blackwell) unlocks larger TP domains. L40S and RTX PRO 6000 have no NVLink at all — they are PCIe-only single-GPU cards. Multi-GPU on those platforms is InfiniBand or Ethernet only.
4. TDP jumped. A100 SXM = 400W. H100 SXM = 700W. B200 SXM ≈ 1kW. GB300 NVL72 is fully liquid-cooled. L40S = 350W (air-cooled in a standard 4U server). RTX PRO 6000 = 600W (air or liquid).
5. MIG isn't universal. H100, H200, B200, GB200, RTX PRO 6000 (up to 4 instances), and A100 support MIG. L40S does not — use time-slicing or vGPU instead.

2.2 What changed Blackwell (2024 → 2026)¶

Verified against the Blackwell architecture page.³

One vendor claim to treat with skepticism: NVIDIA's "30× faster real-time trillion-param inference vs H100" for GB200 NVL72 is from a specific workload (1.8T MoE, FTL=5s) and only achievable inside the NVL72 domain. On a 8×B200 HGX, the speedup over 8×H100 is more like 5–8× for typical 70B-class workloads. Marketing numbers are real but not universally applicable.

2.3 AMD Instinct line-up (CDNA 3 / 4)¶

Verified against AMD's MI300X spec sheet via Wikipedia cross-reference.⁹

The MI300X's killer feature is the 192 GB HBM3. That's why it became the de-facto "run a 70B model in FP16 on one GPU" chip in 2024 — the H100's 80 GB couldn't. The trade-off: ROCm software maturity still trails CUDA by ~6 months. vLLM runs on ROCm but feature parity lags.

CDNA 4 (MI350X, June 2025) is AMD's first accelerator with native FP4 — and on the MI355X, that's 9.2 PFLOPS of FP4 (sparse). Same conversation as Blackwell B200 on raw compute; software ecosystem question remains.

2.4 The interconnect constraint¶

LLM serving has two distinct network planes:

The hard rule: tensor-parallel only works within a single NVLink domain. If you need 16 GPUs in one TP group, they must be in one server (or one NVL72 rack). Crossing to InfiniBand forces pipeline-parallel or data-parallel. A "pool of GPUs" is therefore not uniform — see §7 for the practical impact.

Hopper's DGX GH200 supported 256 H100s in a single NVLink domain (57.6 TB/s all-to-all). Blackwell NVL72 does 72 B200s in one domain with 130 TB/s. Rubin is targeting 576 GPUs in one NVLink domain.

3. Virtualization layer: passing GPUs to VMs¶

3.1 GPU passthrough (VFIO/IOMMU)¶

The hypervisor exposes the PCIe device to the guest; the guest loads the NVIDIA driver directly; CUDA calls run as if on bare metal. Highest-performance GPU virtualization option. Standard for enterprise LLM VMs.

Five gotchas that will bite you:

1. Reset bug. NVIDIA consumer GeForce cards do not support function-level reset (FLR) on most PCIe root complexes. After VM shutdown, the GPU is stuck in a bad state and the host can't reclaim it. Workarounds: vendor-reset kernel module for older cards, or H100/A100 data center SKUs which do support reset. For Hopper/Blackwell data center parts this is a non-issue; for any GeForce/RTX card in passthrough, it is.
2. MIG + passthrough is unsupported. MIG instances are hardware partitions; you can't pass a single MIG instance to a VM via VFIO. If you need MIG isolation, configure it at the host level (Kubernetes device plugin MIG mode) and expose to a container, not a VM.
3. vGPU vs passthrough. vGPU (NVIDIA's licensed product, §3.2) supports MIG-like time-slicing with hypervisor integration, and can give a fraction of a MIG to a VM. For LLM inference in a multi-tenant cloud, this is the standard play.
4. Memory-mapped I/O regions. Modern GPUs have BARs > 4 GB. You need UEFI/BIOS with Above 4G Decoding enabled, and the kernel booted with iommu=on. Some Hyper-V and older VMware versions don't support 64-bit BARs.
5. Confidential computing on GPUs is a 2025+ story. Hopper introduced TEE-mode; Blackwell added TEE-I/O (encryption between GPUs). For VMs this means the guest can use the GPU without the hypervisor seeing plaintext, but the implementation only works on bare metal (no hypervisor in the data path) with specific driver versions.⁶

3.2 The six ways to share a GPU¶

3.3 The MIG profile table that matters¶

Verified against NVIDIA's MIG tech page (2026-06-16).⁷⁸

The "g" suffix is the compute slice; the GB suffix is the memory slice. You can't mix 1g and 3g on the same GPU unless using mig.config to specify a heterogeneous layout (supported on A100+).

For LLM inference: the 1g.10gb profile on H100 is useless for production LLM. Even a 7B model in FP16 with KV cache wants ~20 GB. The 3g.40gb profile is the practical minimum for serving a 7B/13B model with reasonable batch. The 4g.40gb profile is the workhorse for multi-tenant LLM serving.

A common mistake: enabling MIG on H100 and trying to use 1g.10gb for Llama-7B. You'll OOM on KV cache within seconds under any real load.

3.4 Bare metal vs VM for LLM serving¶

Most production LLM serving shops run bare metal: - Latency: no hypervisor context-switch overhead - Full memory: VM ballooning or KSM can interfere with CUDA's huge-page allocation - MIG is easier to configure at the host - vGPU license cost: ~$1,000+ per GPU per year

VMs are common in: - Enterprise on-prem where IT mandates virtualization - Multi-tenant clouds where each tenant must be in its own VM (security/compliance) - GPU-sharing scenarios where vGPU's licensing model is more flexible than MIG - Mainstream enterprise deployments on L40S or RTX PRO 6000 — both have first-class vGPU support and are the de-facto "VDI + AI" cards for non-cloud deployments²¹²²

3.5 GPUDirect RDMA (for multi-node training)¶

NCCL uses GPUDirect RDMA to send GPU memory directly to a remote NIC without bouncing through host CPU. Requires:

• RDMA-capable NIC (ConnectX-6/7/8, BlueField-3)
• Peer-to-peer driver (nvidia-peermem, Mellanox peer memory)
• NCCL with NCCL_IB_HCA and NCCL_SOCKET_IFNAME properly set
• InfiniBand or RoCE fabric

In K8s this needs Multus CNI to attach the RDMA NIC as a secondary interface, plus SR-IOV CNI or host-device CNI. The classic gotcha: NCCL on a default K8s pod cannot see IB devices — the IB userspace drivers must be in the container image AND the pod needs access to the device files. This is one of the top reasons "training on K8s is hard."

4. Container layer: what nvidia.com/gpu: 1 actually means¶

4.1 The OCI hook chain¶

When you run a container with --gpus all (Docker) or nvidia.com/gpu: 1 (K8s):

1. NVIDIA Container Runtime (nvidia-container-runtime) intercepts the container create request. It's an OCI runtime, not a high-level runtime.
2. The runtime invokes the prestart hook nvidia-container-cli.
3. nvidia-container-cli queries GPU(s) on the host via NVML, looks up the requested devices, generates a CDI (Container Device Interface) spec or a set of bind mounts + env vars.
4. The container is created with:
5. NVIDIA driver files bind-mounted from the host (/usr/lib/x86_64-linux-gnu/libcuda.so.*, etc.)
6. CUDA libraries injected
7. GPU device files passed through (/dev/nvidia0, /dev/nvidiactl, /dev/nvidia-uvm, /dev/nvidia-uvm-tools)
8. NVIDIA_VISIBLE_DEVICES set in env

Inside the container, the process sees a normal Linux system with a working CUDA stack. It cannot tell it isn't bare metal.

Source: NVIDIA device plugin README and Container Toolkit install guide.¹⁰¹¹

4.2 Two ways to expose devices¶

Legacy NVIDIA_VISIBLE_DEVICES env var: - The runtime sets NVIDIA_VISIBLE_DEVICES=0,1 in the container - The runtime generates a CDI spec at runtime that includes the corresponding devices - Same as the --gpus device=0,1 Docker syntax

CDI (Container Device Interface) — newer, CNCF-incubating: - The operator runs nvidia-ctk cdi generate on the host to produce a CDI spec file in /etc/cdi/ - The container runtime reads the CDI spec and applies it - More structured, supports any device (not just NVIDIA), the direction the K8s ecosystem is moving

The k8s-device-plugin v0.19.x supports both. For new deployments, the recommendation is CDI — but legacy mode still works.¹²

4.3 What you need on the host¶

For a container to use a GPU, the host must have:

1. NVIDIA kernel driver (nvidia.ko, nvidia-uvm.ko, nvidia-modeset.ko)
2. NVIDIA userspace driver (libcuda.so, libnvidia-*.so)
3. nvidia-container-toolkit (provides the OCI hook)
4. containerd/CRI-O/Docker configured to use nvidia-container-runtime (either as default or via RuntimeClass)
5. The GPU device files (/dev/nvidia*)

If you build the host with a custom AMI that bakes all this in, you're done. If you want a generic OS image and let K8s manage it, you want the GPU Operator (§5.2).

5. Kubernetes layer: the practical reality¶

5.1 The device plugin model (legacy, still dominant in 2026)¶

The nvidia-device-plugin (k8s-device-plugin) runs as a DaemonSet on GPU nodes.¹⁰ Latest: v0.19.2 (released 2026-05-27).

Flow: 1. Plugin queries nvidia-smi on startup to count GPUs (and MIG instances if MIG mode is enabled) 2. Registers GPUs with kubelet as extended resources: nvidia.com/gpu=N (or nvidia.com/mig-1g.5gb=N) 3. Pods request GPUs with resources.limits.nvidia.com/gpu: 1 4. Kubelet calls the plugin's Allocate RPC during admission 5. Kubelet passes device IDs to the container runtime, which invokes nvidia-container-cli with NVIDIA_VISIBLE_DEVICES=<those ids>

What it does NOT do well: - Health checking: the plugin only knows "GPU was there at registration time." If a GPU dies, the pod keeps running with a missing device. - Cleanup: if a pod is killed ungracefully, the GPU may stay "allocated" until the kubelet restarts the plugin. - Heterogeneous allocation: nvidia.com/gpu: 2 works, but you can't say "give me 80GB of memory and 200 GB/s of bandwidth from any of these 3 GPU types."

DRA (§5.3) was designed to address these gaps.

5.2 GPU Operator: the full stack¶

If you want the "just works" path, install the GPU Operator. Latest: v26.3.2 (released 2026-05-29).¹³

It manages the entire stack as Kubernetes objects:

What's new in v26.3.2 (verified against release notes): - Kubernetes 1.36 support added - NRI Plugin support for CRI-O 1.34+ - DCGM Exporter can include pod labels/UIDs in metrics (high-cardinality risk: set podLabelAllowlistRegex!) - Custom annotations on DCGM Exporter DaemonSet - DRA driver is on the roadmap, not yet integrated

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
helm install --wait --generate-name \
    -n gpu-operator --create-namespace \
    nvidia/gpu-operator \
    --set driver.enabled=true

Why the GPU Operator is required for Karpenter GPU nodes: a Karpenter-provisioned p5.48xlarge is, at boot, a stock AWS AMI with no NVIDIA driver, no container toolkit, no device plugin. The pod requesting nvidia.com/gpu: 1 will be stuck Pending forever because the node has no nvidia.com/gpu extended resource to advertise. The Operator's driver DaemonSet installs the matching nvidia.ko for the node's kernel, the device plugin starts, and within ~3–5 minutes the node advertises nvidia.com/gpu: 8.

Skip the Operator only if you're using a custom AMI with the driver pre-baked (AWS DLAMI, GCP Deep Learning VM) AND you install the device plugin DaemonSet yourself.

5.3 DRA (Dynamic Resource Allocation): the future¶

KEP-3094 introduced DRA as a generalized replacement for the device plugin model. Key primitives:

• ResourceClaim — a namespaced handle to allocated devices. The pod references it; the scheduler gives it to the pod.
• ResourceClaimTemplate — a template for creating claims dynamically per pod.
• ResourceSlice — a cluster-scoped object describing a pool of devices. DRA drivers create and manage these.
• DeviceClass — a cluster-scoped object defining a class of device (e.g., "consumer GPU", "H100 with at least 80GB").

Kubernetes 1.36 status (verified against the DRA docs 2026-06-16):¹⁴

Bottom line for 1.36: the LLM-relevant DRA features (partitionable devices, consumable capacity, device binding conditions) are all beta and default-on. The alpha leftovers (node allocatable resources, device metadata in containers, list type attributes) are for advanced cases.

Critical caveat — the 1.36.2 fix: K8s 1.36.0/1 had a critical double-allocation bug in DRA (#139211) affecting drivers using SharedCounters + DRAConsumableCapacity. Could cause workload failures, device conflicts, crashes, or data loss. If you use DRA, run on 1.36.2+ (released 2026-06-12).¹⁵

The NVIDIA DRA Driver for GPUs (latest: v0.4.0, released 2026-05-15) implements two sub-plugins:¹⁶

1. GPU kubelet plugin — handles GPU allocation. Currently experimental, not yet officially supported. Helm chart leaves it disabled by default. Use only for exploration.
2. ComputeDomain kubelet plugin — manages Multi-Node NVLink (MNNVL) across pods. Officially supported. Wires IMEX (Inter-Memory Exchange) daemons, domains, and channels under the hood. This is the K8s-native orchestration of NVL72-style domains — what you need to use a 72-GPU NVLink domain as a single scheduling unit for trillion-param models.

5.4 GPU sharing in K8s: three modes¶

Mode 1: Exclusive (the default)

resources:
  limits:
    nvidia.com/gpu: 1

Plugin gives the pod one entire GPU. Best for memory-bandwidth-bound decode, high-throughput training.

Mode 2: Time-slicing (free, no isolation)

# ConfigMap: nvidia-device-plugin-config
data:
  config.yaml: |
    version: v1
    sharing:
      timeSlicing:
        renameByDefault: false
        failRequestsGreaterThanOne: false
        resources:
          - name: nvidia.com/gpu
            replicas: 4   # advertise each H100 as 4 GPUs

With replicas: 4 on 8-GPU p5.48xlarge, you can run 32 vLLM pods on the same hardware. Caveat: no isolation. If two pods both try to do FP16 matmul, they contend for SM cycles. Throughput degrades 30-50% under contention. Works for batch=1 inference, fails for sustained high-batch throughput.

The 30B-A3B-at-128K trap: each pod needs ~70 GB of GPU memory. 4× replicas of a 30B-A3B pod on one H100 = 280 GB demanded, more than 80 GB available. Time-slicing is for small models (7B-class) or short contexts. Don't try to host 30B-A3B in 4 time-sliced replicas on one H100.

Mode 3: MIG (hardware isolation, A100/H100 only)

# ClusterPolicy for GPU Operator
apiVersion: nvidia.com/v1
kind: ClusterPolicy
metadata:
  name: gpu-cluster-policy
spec:
  migManager:
    enabled: true
  mig:
    strategy: mixed   # resources become nvidia.com/mig-1g.5gb etc.

With strategy: mixed, you get resources like nvidia.com/mig-3g.40gb (3/7 of an H100) that pods can request. MIG reconfigures take 30–60 seconds (GPU goes offline briefly) — MIG is for stable partitioning, not dynamic multi-tenancy. Right answer for SLA-bound multi-tenant inference.

5.5 The driver-upgrade gotcha¶

The GPU Operator manages the kernel driver. When you upgrade (e.g., to pick up a security fix):

1. Operator cordons the node
2. Drains running GPU pods (respects PodDisruptionBudgets)
3. Restarts the driver pod (kernel module reload)
4. Restarts the device plugin, MIG manager, etc.
5. Uncordons

This disrupts running GPU pods. Watch the nvidia.com/gpu-driver-upgrade-state node label — value upgrade-done means the node is past the upgrade; upgrade-in-progress is the time window where the node has no GPU to allocate. If you cordon a node manually while the operator is already upgrading, the operator will uncordon and proceed — don't race it.

The pattern that avoids the disruption: upgrade via node replacement, not in-place. Karpenter with a new GPU NodePool spins up a node with the new driver, drains the old one, terminates it. Application sees a brief connection blip, not a pod restart.

5.6 Observability: the must-have metrics¶

The DCGM Exporter (latest: 4.5.3-4.8.2, released 2026-05-07) runs as a sidecar on GPU nodes, exposing Prometheus metrics on port 9400.¹⁷ Install via GPU Operator (it manages DCGM Exporter as part of the stack).

6. Terminology clarification: "passthrough" means different things¶

When a post says "GPU passthrough," which one does it mean?

"Do I need to set GPU passthrough?" No — you don't "set" it. nvidia.com/gpu: 1 in the pod spec is passthrough. The system needs: NVIDIA driver on the node, NVIDIA Container Toolkit as containerd runtime, and the k8s-device-plugin DaemonSet advertising the resource. GPU Operator bundles all three plus MIG manager, DCGM exporter, feature discovery, and the DRA driver.

7. Multi-GPU pooling for large LLMs¶

This is the core question. The answer has three layers: hardware constraint → software stack → workload pattern.

7.1 The fundamental hardware constraint¶

NVLink topology is not uniform. On a single 8-GPU HGX H100 baseboard: - 8 GPUs in a fully connected NVLink Switch fabric (900 GB/s all-to-all) - They can do tensor-parallel efficiently up to 8 GPUs

To go to 16 GPUs for one model: - You need a second server with NVLink bridge (rare) OR - Accept pipeline-parallel across InfiniBand (slower but works) OR - Scale out to a GH200/NVL72 domain (much more expensive)

In practice: most production LLM deployments never put more than 8 GPUs in one tensor-parallel group. Beyond that, pipeline-parallel or data-parallel takes over.

7.2 The workload patterns¶

For inference, TP is the king of latency — but it has the most expensive inter-GPU bandwidth requirements. For throughput-oriented serving (batch APIs, high QPS), DP wins on cost-per-token. For training, you typically combine: TP for memory + compute efficiency within a node, PP across nodes, DP for further scaling.

7.3 The memory math (worked example)¶

For a 70B model in FP16 (FP16 = 2 bytes per parameter, the standard precision for serving): - Weights: 70B × 2 bytes = 140 GB - Doesn't fit on a single 80 GB H100

TP2 across 2× H100 (80 GB each): (TP2 = tensor-parallel across 2 GPUs) - Weights: 70 GB per GPU - KV cache per token (Llama-70B arch): 2 × 80 layers × 64 heads × 128 head_dim × 2 bytes = ~2.6 MB/token¹⁸ (KV cache = the running memory of attention, see Glossary) - For 8k context, 32 concurrent users: 256k tokens × 2.6 MB = ~665 MB per GPU for KV cache - Per-GPU: 70 GB + 665 MB ≈ 71 GB ✓ (fits in 80 GB)

TP4 across 4× H100 (80 GB each): - Weights: 35 GB per GPU - KV cache: same ~665 MB per GPU - Per-GPU: 35.7 GB ✓ (leaves room for activations and batch)

TP72 in a GB200 NVL72 (one rack): - A 1.8T MoE model (the GB200 NVL72's benchmark target) can use TP=72 in one domain — first time this is possible at full bandwidth

Sources: vLLM's prefix caching docs for KV cache block management; HuggingFace model memory anatomy for the bytes-per-param math.¹⁸¹⁹

7.4 The K8s primitives for "GPU pool"¶

There are three abstractions to understand:

1. GPU Operator — gives you GPUs-as-a-resource on nodes. Static pool.
2. Karpenter / Cluster Autoscaler — adds/removes nodes based on pending GPU pods. Dynamic pool.
3. DRA + ComputeDomain — the future. Lets you say "give me 16 GPUs in one NVLink domain" and the scheduler finds them.

For LLM workloads specifically:

• Single-tenant, predictable load: Static pool of GPU nodes with the GPU Operator. Don't bother with auto-scaling. Add capacity by adding nodes.
• Bursty inference load: Karpenter with GPU NodePool. New nodes are vanilla AMI; GPU Operator installs the driver on them. vLLM pods start. Use karpenter.sh/v1 NodePool + karpenter.k8s.aws/v1 EC2NodeClass.
• Multi-tenant, MIG-partitioned: GPU Operator with MIG mode enabled. Each pod gets a MIG instance. Kueue for fair scheduling.
• Large models across nodes: PipelineParallel with vLLM v0.23.0 or TensorRT-LLM. Need InfiniBand. Need NCCL env vars baked into the pod.

7.5 The Kubeflow Trainer piece (for training)¶

If you're training LLMs on K8s, Kubeflow Trainer (latest: v2.2.0, released 2026-03-20) is the de-facto orchestrator. It provides:

• TrainJob CRD — declare a distributed training job
• PyTorch Runtime, MPI Runtime, JAX Runtime — pick the framework
• JobSet and LeaderWorkerSet integration — for gang scheduling
• Kueue integration — for topology-aware scheduling
• Distributed data cache — streams training data to GPU nodes without copying through CPU

For multi-node training, the canonical setup is: - Kubeflow Trainer with PyTorch Runtime - Each worker pod has: - GPU requests (via device plugin or DRA) - RDMA device request (via SR-IOV CNI + sriov-device-plugin) - Env vars: NCCL_IB_HCA=mlx5, NCCL_SOCKET_IFNAME=eth0, NCCL_DEBUG=INFO - Shared memory volume (/dev/shm) sized large enough for DDP - The NCCL debug volume mounted so you can read /tmp/nccl_log after the job

8. Decision framework¶

8.1 "Which GPU do I buy?"¶

8.2 "How do I share a GPU?"¶

8.3 "How do I run multi-GPU inference?"¶

8.4 "How do I provision GPU nodes on K8s?"¶

9. Best practices distilled¶

1. For production LLM serving on Hopper/Blackwell: use the GPU Operator. Don't roll your own driver+toolkit+plugin stack. The operational pain of upgrading drivers in a vanilla cluster is not worth it.
2. For inference, default to TP2 or TP4 within a node. TP8 is sometimes needed for very large models, but the latency improvement of going from TP4 to TP8 is small relative to the cost. TP16+ is almost always wrong unless you're in a single NVLink domain (NVL72).
3. For multi-tenant inference, use MIG on H100/H200/B200. Time-slicing is a footgun — when one tenant pushes SMs to 100%, the others tank. MIG gives you hard isolation.
4. For training, use Kubeflow Trainer with PyTorch Runtime. Don't try to roll your own multi-node job controller. Kueue for topology-aware scheduling.
5. For very large models (200B+): plan your NVLink topology before buying. If you need TP16, you need NVL72 (or wait for Rubin). Don't assume "we have 64 GPUs, we can serve a 200B model" — it depends on the interconnect.
6. Always run DCGM Exporter and alert on the right metrics. ECC errors >0, temperature >85°C sustained, framebuffer <1 GB free.
7. MIG profile selection is workload-driven. Don't use 1g.10gb for LLMs. The smallest profile you should consider for production LLM is 3g.40gb on H100.
8. DRA is the future but not yet the default. As of K8s 1.36, the basic DRA primitives are beta+enabled-by-default. The NVIDIA DRA driver is at v0.4.0. Most production clusters are still on the device plugin model in 2026. Plan a migration, but don't block on it. And never run DRA on K8s 1.36.0 or 1.36.1 — the double-allocation bug is a footgun.
9. For multi-node training, NCCL is the hidden landmine. Bake the env vars, the device files, the IB driver, and the shared memory volume into your pod spec. Don't try to make it "just work" — be explicit.
10. For AMD MI300X/MI350X: validated for inference (vLLM ROCm build works), and AMD has the 192GB / 288GB advantage for single-GPU fitting of large models. Software maturity still trails NVIDIA; budget for ROCm debugging time.

10. Gotchas and "this will bite you" list¶

• Vendor reset for consumer GPUs (§3.1) — if you try to use GeForce in production, plan for this.
• MIG + VM passthrough is unsupported — MIG is a host-level concept.
• 1g MIG profile is too small for LLMs (§3.3).
• Default K8s pods can't see IB devices (§3.5) — needs Multus + SR-IOV CNI.
• GPU Operator upgrades disrupt running pods (§5.5) — schedule maintenance windows or use node replacement.
• nvidia.com/gpu resource is integer-counted — no fractional allocation (use MIG for that).
• DRA DRANodeAllocatableResources is alpha in 1.36 — don't rely on it yet.
• DRA device plugin is not yet officially supported for basic GPU allocation — NVIDIA explicitly says "experimental, not yet officially supported" in their v0.4.0 README. ComputeDomain is officially supported.
• CDI adoption is uneven — containerd ≥ 1.7, CRI-O ≥ 1.24. Older versions still use the legacy NVIDIA_VISIBLE_DEVICES path.
• ROCm vs CUDA parity — when comparing AMD MI300X to H100, don't assume feature parity in the serving stack. vLLM ROCm builds lag CUDA by ~6 months.
• Blackwell requires specific driver versions — CUDA 12.8+ for B200, 12.9+ for B300. Older CUDA will fail to load.
• "FP4 is good enough" is still an open research question. NVFP4 has accuracy trade-offs that depend on the model. Don't deploy FP4 in production without measuring on your workload.
• Time-slicing + 30B-A3B at 128K context won't fit — 4× replicas of a 30B-A3B pod on one H100 demand 280GB, more than 80GB available. Time-slicing is for small models or short contexts.

Summary¶

• Physical: LLM workloads are memory-bandwidth-bound during decode and topology-bound when multi-GPU. HBM3e + NVLink bandwidth are the metrics that matter, not raw TFLOPS. H200 (141 GB, 4.8 TB/s) is the workhorse for 70B single-GPU inference. B200/B300 + NVL72 unlock trillion-param at rack scale.
• VM: Bare metal for production LLM serving (latency, MIG, license cost). VMs for multi-tenant cloud or compliance-bound enterprise. The 1g MIG profile is too small for LLMs — use 3g or 4g minimum.
• Container: nvidia.com/gpu: 1 IS the K8s-level "passthrough" — there is nothing to "set." nvidia-container-runtime mounts the device files; nvidia-ctk cdi generate is the modern path (CDI). GPU Operator bundles all of it.
• K8s: Device plugin (v0.19.2) is the legacy default. DRA is the future — beta in K8s 1.36, but use 1.36.2+ to avoid the double-allocation bug. NVIDIA DRA driver v0.4.0 supports ComputeDomain (MNNVL) officially; GPU allocation plugin is experimental.
• Multi-GPU: TP2/TP4 within a node is the latency sweet spot. TP8 maxes out a single NVL8. Beyond 8 GPUs, switch to PP+IB or NVL72-scale. MoE = expert-parallel + all-to-all bandwidth is the constraint.
• Operations: DCGM Exporter for metrics (DCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_FB_FREE, DCGM_FI_DEV_XID_ERRORS). GPU Operator upgrades disrupt pods — replace nodes via Karpenter, don't upgrade in place.

Versions referenced (verified 2026-06-16)¶

Confidence & gaps¶

• Strong primary-source support: every version in the table, every MIG profile size, every DCGM metric, all 4 layers' tooling. K8s 1.36 DRA status verified against the K8s docs feature-gate table.
• Sources disagreed on: the vGPU license cost (industry-typical, ~$1K+/GPU/year, but I couldn't pin a current price). Marked as "industry-typical" in §3.4.
• Couldn't verify / illustrative: vLLM ROCm feature-parity delta (anecdotal community reports; not pinned to a primary source). K8s 1.36.0 release date — the knowledge bank says "April 2026" approximately; I cite 1.36.2 (2026-06-12) and feature status which is what the docs confirm.
• Vendor marketing claims flagged: NVIDIA's "30× faster real-time trillion-param inference" is for a specific workload (1.8T MoE, FTL=5s) on the NVL72 domain — not universally applicable. Marked as such in §2.2.

Questions or discussion? Connect on LinkedIn, X or reach out via email.

Discussion

Have thoughts on this post? Share them below — questions, corrections, or your own experience are all welcome.