GPUs on Kubernetes: From Bare Metal to Schedulable in One Operator¶

A fresh Kubernetes cluster has no idea your nodes have GPUs. kubectl describe node shows CPU, memory, and pods — nothing else. To make a pod request a GPU you need a driver, a container runtime hook, and a device plugin advertising the hardware to the scheduler, all version-matched across every GPU node. Do it by hand and you'll re-do it on every kernel bump. This post wires it up the way you actually want — one operator — and works on any Kubernetes, not a specific vendor's distro.

Verified against NVIDIA GPU Operator v26.3.x, driver branch 570/580, CUDA 13.2, and Kubernetes 1.33/1.34 (June 2026). This is the enablement layer — the foundation the rest of the GPU series builds on:

• Sizing — GPU for LLM workloads reference: which card, how much VRAM.
• Sharing, compared — Time-slicing vs MIG vs MPS: which mode, and why.
• Sharing, the recipe — Stacking MIG + time-slicing in one values.yaml: the exact config, wired through GitOps.
• The platform — Self-hosted token-as-a-service: the OpenAI-compatible inference gateway you run on top.

What a GPU node actually needs¶

Kubernetes schedules cpu and memory natively. A GPU is an extended resource — the cluster only knows it exists if something advertises it. That something is a chain of four pieces on every GPU worker:¹

Build that by hand and you own driver/CUDA/kernel compatibility forever. The GPU Operator packages all of it as containers managed by a single Helm release, reconciled by a controller. New GPU node joins, gets labeled, gets the whole stack — zero touch.²

Install the GPU Operator on any cluster¶

Works on upstream kubeadm, k3s, RKE2, EKS, GKE, AKS — anywhere you have kubectl and helm. The only real prerequisite is that nodes have NVIDIA GPUs and a supported OS.

Prerequisites:

• Kubernetes 1.29+ (1.34+ if you want DRA — see below).
• helm 3.x and cluster-admin.
• Don't pre-install the NVIDIA driver or toolkit on GPU nodes — let the operator own them (or explicitly tell it they're pre-installed). Mixing the two is the #1 install failure.
• containerd or CRI-O as the runtime.

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

helm install --wait gpu-operator \
  -n gpu-operator --create-namespace \
  nvidia/gpu-operator --version v26.3.0

That's the whole install. The operator deploys Node Feature Discovery, then rolls out the driver, toolkit, device plugin, GFD, and DCGM telemetry only on nodes that have GPUs. Watch it converge:

kubectl get pods -n gpu-operator

NAME                                       READY   STATUS      RESTARTS   AGE
gpu-feature-discovery-kqqnv                1/1     Running     0          90s
gpu-operator-...                           1/1     Running     0          3m
nvidia-container-toolkit-daemonset-jqklh   1/1     Running     0          2m
nvidia-cuda-validator-nwtms                0/1     Completed   0          60s
nvidia-dcgm-exporter-2bb9k                 1/1     Running     0          90s
nvidia-device-plugin-daemonset-dxskt       1/1     Running     0          90s
nvidia-driver-daemonset-kfc2c              1/1     Running     0          2m
nvidia-operator-validator-wwx7g            1/1     Running     0          80s

The validator pods are the operator proving each layer works before marking the node ready. If nvidia-operator-validator is Running, GPUs are schedulable.

Already have a driver on the host?¶

Bare-metal shops often bake the driver into the node image. Tell the operator to skip it:

helm install gpu-operator -n gpu-operator --create-namespace \
  nvidia/gpu-operator --version v26.3.0 \
  --set driver.enabled=false

The operator then manages everything except the driver. Pick one owner for the driver and stick with it.

Prove a GPU is schedulable¶

Don't trust the install log. Check that the node advertises the resource:

kubectl describe node <gpu-node> | grep -A6 Allocatable

Allocatable:
  cpu:                8
  memory:             65765624Ki
  nvidia.com/gpu:     4
  pods:               110

nvidia.com/gpu: 4 is the win — the scheduler now knows this node has 4 GPUs. Run a throwaway pod that requests one:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-smoke-test
spec:
  restartPolicy: Never
  containers:
    - name: cuda
      image: nvcr.io/nvidia/cuda:13.2.0-base-ubuntu24.04
      command: ["nvidia-smi"]
      resources:
        limits:
          nvidia.com/gpu: 1     # request is implied; GPUs aren't overcommittable

kubectl apply -f gpu-smoke-test.yaml
kubectl logs gpu-smoke-test

You should see the nvidia-smi table with the card, driver version, and CUDA version. A GPU limit can't be overcommitted — one pod holds the whole device until it exits. That's exactly the problem the next section fixes.

Target a specific GPU model¶

GFD labels every node, so you can pin a pod to the hardware it needs instead of gambling on the scheduler:

  nodeSelector:
    nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3

kubectl get nodes -l nvidia.com/gpu.present=true --show-labels lists what you've got.

One GPU, many pods: sharing strategies¶

By default one pod equals one whole GPU. For an 80GB H100 running a tiny model or a notebook, that's a waste. Three ways to share, each a different trade between isolation and cost:³

The two short examples below get you sharing fast. For the full decision treatment — when each mode wins, the trade-offs in depth — see Time-slicing vs MIG vs MPS, compared. To stack MIG and time-slicing together with a production values.yaml applied through GitOps, see Stacking MIG + time-slicing.

Time-slicing — quickest win¶

Tell the device plugin to advertise each physical GPU as N virtual ones. Apply a ConfigMap and point the operator at it:

apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
  namespace: gpu-operator
data:
  any: |-
    version: v1
    sharing:
      timeSlicing:
        resources:
          - name: nvidia.com/gpu
            replicas: 4      # one GPU now advertises as 4

kubectl patch clusterpolicy/cluster-policy --type merge \
  -p '{"spec":{"devicePlugin":{"config":{"name":"time-slicing-config","default":"any"}}}}'

nvidia.com/gpu on the node jumps from 1 to 4. Four pods land on one card. No memory isolation — one greedy pod can OOM the others. Fine for dev, dangerous for tenants who don't trust each other.

MIG — hardware isolation on A100/H100/H200/B200¶

MIG carves a single GPU into up to 7 fully isolated instances, each with its own SMs and memory slice. The operator's MIG Manager does it from a node label:⁴

# Single strategy: every instance the same size, advertised as nvidia.com/gpu
kubectl label node <gpu-node> nvidia.com/mig.config=all-1g.10gb --overwrite

After it reconfigures, the node advertises 7 schedulable slices. Use mixed strategy when you want different sizes on one card — the node then advertises typed resources you request explicitly:

nvidia.com/mig-1g.10gb:   4
nvidia.com/mig-2g.20gb:   1
nvidia.com/mig-3g.40gb:   1

  resources:
    limits:
      nvidia.com/mig-2g.20gb: 1

This is the only sharing mode with real fault and performance isolation — a crash or a memory hog in one instance can't touch another. That's why it's the default for multi-tenant clusters.

The 2026 shift: Dynamic Resource Allocation (DRA)¶

The nvidia.com/gpu: 1 model has a ceiling: it's a counter. You can't say "give me a GPU with at least 40GB free, shared with my other pod, configured for MPS" in a Pod spec. DRA fixes that, and it went GA in Kubernetes 1.34.⁵

DRA replaces the count-based request with a claim against a vendor driver, so you describe the device you want and the driver allocates and configures it:

apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
  name: shared-gpu
spec:
  devices:
    requests:
      - name: gpu
        deviceClassName: gpu.nvidia.com
---
apiVersion: v1
kind: Pod
metadata:
  name: dra-workload
spec:
  resourceClaims:
    - name: gpu
      resourceClaimName: shared-gpu
  containers:
    - name: cuda
      image: nvcr.io/nvidia/cuda:13.2.0-base-ubuntu24.04
      resources:
        claims:
          - name: gpu

Two pods referencing the same ResourceClaim share one physical GPU — declaratively, without a cluster-wide time-slicing ConfigMap. It also handles things the device plugin never could: multi-node NVLink domains (ComputeDomains) for GB200-class systems, and per-workload sharing config.

When to adopt it:

• Stay on the device plugin if you're on K8s < 1.34 or your sharing needs are simple (whole GPUs, basic time-slicing, MIG). It's stable and battle-tested.
• Move to DRA for fine-grained sharing, mixed workloads on one card, or NVLink-domain scheduling. Needs K8s 1.34.2+, driver 580+, CDI enabled in the runtime, and the NVIDIA DRA driver (GPU Operator v25.10+ with the classic device plugin disabled to avoid conflicts).

Don't run both allocators against the same GPUs. Pick the device plugin or DRA per node pool.

Telemetry you get for free¶

The operator ships DCGM Exporter as a DaemonSet, exposing per-GPU metrics in Prometheus format — utilization, memory, temperature, power, ECC errors — mapped back to the pod using each GPU.⁶

kubectl -n gpu-operator port-forward ds/nvidia-dcgm-exporter 9400:9400
curl -s localhost:9400/metrics | grep DCGM_FI_DEV_GPU_UTIL

If you already run Prometheus (the kube-prometheus-stack), add a ServiceMonitor and import Grafana dashboard 12239 (NVIDIA DCGM Exporter) for a ready-made GPU view. The metric that matters most isn't temperature — it's DCGM_FI_DEV_GPU_UTIL next to your spend. GPUs idling at 10% are the most expensive line in the cluster. This is the same Prometheus/Grafana pattern from the Kong and ingress-logging posts, just pointed at silicon.

Day-2: upgrades and the gotchas that bite¶

• Upgrade via Helm, but apply the new CRDs first — helm upgrade won't update CRDs on its own. Driver upgrades drain and reboot GPU nodes, so respect PodDisruptionBudgets and cordon during maintenance windows.
• Kernel updates break unsigned drivers. If you patch the node OS and the kernel moves, the driver container rebuilds against the new headers. On Secure Boot nodes you need signed modules or pre-compiled driver images, or the driver pod will CrashLoopBackOff.
• Never let the host driver and the operator driver coexist. Symptom: nvidia-smi works on the host but pods can't see the GPU, or the driver DaemonSet won't start. Pick one.
• nvidia-operator-validator stuck? It's the canary. kubectl logs it — usually a toolkit/runtime mismatch or a node missing the nvidia runtime class.
• MIG changes are disruptive. Reconfiguring MIG geometry evicts everything on that GPU. Drain first; don't relabel a busy node.

Summary¶

Turn a blank cluster into a GPU cluster, in order:

• Install the GPU Operator with one Helm release; let it own driver, toolkit, device plugin, and GFD. Don't pre-install the driver unless you tell the operator.
• Verify nvidia.com/gpu shows in kubectl describe node, then run an nvidia-smi smoke-test pod.
• Target hardware with GFD labels (nvidia.com/gpu.product) instead of hoping the scheduler picks right.
• Share GPUs by need: time-slicing for dev, MPS for many small inferences, MIG for production multi-tenant isolation.
• Adopt DRA on K8s 1.34+ for declarative sharing and NVLink-domain scheduling; otherwise the device plugin is the stable default. Never run both on the same nodes.
• Wire DCGM Exporter to Prometheus and watch DCGM_FI_DEV_GPU_UTIL against cost — idle GPUs are the budget leak.
• Plan day-2: CRDs before Helm upgrade, signed drivers on Secure Boot, drain before MIG changes, one driver owner.

The whole job is making nvidia.com/gpu (or a DRA claim) appear in a Pod spec and actually run. The operator turns a fragile, per-node driver dance into a single reconciled release — which is exactly what you want before you put expensive silicon behind a scheduler.

Questions or discussion? Connect on LinkedIn, X or reach out via email.

Discussion

Have thoughts on this post? Share them below — questions, corrections, or your own experience are all welcome.