GPU DRA on Kubernetes: migrating off the NVIDIA device plugin, end-to-end¶

Dynamic Resource Allocation went GA in Kubernetes 1.35¹, and NVIDIA's DRA driver for GPUs shipped v0.4.1 on 2026-06-30². The nvidia.com/gpu extended resource model — the one the device plugin has used since 2017 — now has a sanctioned successor. The migration is non-trivial because the resource model is completely different (claim-based, not extended-resource-based), but the payoff is real: structured GPU allocation, native NVLink topology awareness, and a single API for MIG, MPS, time-slicing, and Multi-Node NVLink.

This post is the migration path: install the driver, write the first claim, swap one workload at a time, and run device plugin + DRA side-by-side until you're done.

What changed at the API layer¶

The device plugin exposes GPUs as extended resources on nodes:

apiVersion: v1
kind: Pod
metadata:
  name: train-llm
spec:
  containers:
  - name: trainer
    resources:
      limits:
        nvidia.com/gpu: 2  # implicit: any 2 GPUs on a schedulable node

DRA replaces this with ResourceClaims that you reference explicitly:

apiVersion: v1
kind: Pod
metadata:
  name: train-llm
spec:
  resourceClaims:
  - name: gpu-claim
  containers:
  - name: trainer
    resources:
      claims:
      - name: gpu-claim
---
apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
  name: gpu-claim
spec:
  devices:
    requests:
    - name: gpu
      deviceClassName: gpu.nvidia.com
      count: 2

Three things just happened:

1. The Pod no longer references GPUs by name. It references a claim by name.
2. The claim describes what kind of device it wants (a DeviceClass), not a node selector.
3. The scheduler picks the node based on available matching devices — not the kubelet advertising nvidia.com/gpu: 8 and hoping for the best.

For a single-node workload the difference looks cosmetic. For Multi-Node NVLink, MIG, or GPU sharing across pods, the difference is structural — you can express things in DRA that the device plugin API literally cannot represent.

Prerequisites you can't skip¶

DRA was alpha in K8s 1.26, beta in 1.31, and stable (GA) in 1.35¹. The driver and the cluster must agree on the version:

• Cluster — Kubernetes 1.35 or later. The feature gate DynamicResourceAllocation is enabled by default in 1.35+; in 1.32-1.34 you must enable it explicitly on kube-apiserver, kube-controller-manager, kube-scheduler, and kubelet¹².
• GPU Operator — v25.10.0 or later if you want operator-managed installation. v26.3.3 (2026-06-25) is the current stable³. Note: GPU Operator still defaults to managing the device plugin, not the DRA driver. You have to opt in.
• NVIDIA DRA driver — v0.4.1 (2026-06-30) is the current release. Requires Kubernetes 1.32 or newer².
• NVIDIA Container Toolkit — v1.19.1 or later (the DRA driver pulls it in via Helm chart dependency).
• Helm 3 — for the install.
• GPU nodes — same hardware as before. DRA doesn't change what's on the box; it changes how the cluster advertises it.

Run this before you start:

kubectl version --short
helm version --short
nvidia-smi --query-gpu=driver_version --format=csv,noheader

If nvidia-smi shows the driver and kubectl version shows 1.35+, you have the prerequisites. If you're on 1.32-1.34, you need to enable the feature gate cluster-wide first.

Step 1: Install the NVIDIA DRA driver¶

The chart lives at oci://registry.k8s.io/dra-driver-nvidia/charts/dra-driver-nvidia-gpu²:

helm install dra-driver-nvidia-gpu \
  oci://registry.k8s.io/dra-driver-nvidia/charts/dra-driver-nvidia-gpu \
  --version 0.4.1 \
  --namespace dra-driver-nvidia-gpu \
  --create-namespace

What this installs:

• gpu-kubelet-plugin DaemonSet — disabled by default in the chart because GPU allocation features are still "for exploration and demonstration" per the project README². You opt in by setting gpuResourcesEnabled=true.
• compute-domain-kubelet-plugin DaemonSet — officially supported. This is the part that handles Multi-Node NVLink orchestration via IMEX primitives².

For now, leave gpuResourcesEnabled=false (the chart default) and enable only the ComputeDomain plugin. This is the conservative install path: the DRA control plane is ready, but you won't break any existing pods.

Verify the install:

kubectl -n dra-driver-nvidia-gpu get pods
kubectl get deviceclasses

You should see a DeviceClass named gpu.nvidia.com registered.

Step 2: Leave the device plugin running¶

This is the part most blog posts skip: don't uninstall the device plugin yet. Run both for the migration window. The reasons:

1. Existing workloads with nvidia.com/gpu: N requests will keep working until you migrate them one at a time.
2. If something goes wrong with the DRA path, pods still schedule — you haven't broken production.
3. You can A/B test new workloads in DRA mode before committing.

The two systems don't conflict at the API level (extended resources vs ResourceClaims are separate APIs), but they do consume the same underlying GPU hardware. If you enable GPU allocation in the DRA driver (gpuResourcesEnabled=true) while the device plugin is still advertising GPUs, you can get double-counting — both systems report the same GPU as available, two pods land on it, and they fight over memory.

Keep the device plugin running, GPU allocation disabled in the DRA driver. The migration is "claim-based first, extended-resource-based last."

Step 3: Write your first claim¶

Start with the simplest case — a single container that wants one full GPU:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: single-gpu-claim
spec:
  devices:
    requests:
    - name: gpu
      deviceClassName: gpu.nvidia.com
      count: 1

Apply it, then use it from a pod:

apiVersion: v1
kind: Pod
metadata:
  name: smoke-test
  namespace: gpu-test
spec:
  resourceClaimTemplates:
  - metadata:
      name: gpu-claim
    spec:
      devices:
        requests:
        - name: gpu
          deviceClassName: gpu.nvidia.com
          count: 1
  containers:
  - name: cuda-app
    image: nvcr.io/nvidia/k8s/cuda-sample:nbody-cuda12.5.0
    resources:
      claims:
      - name: gpu-claim
    command: ["/cuda-sample/nbody"]

A ResourceClaimTemplate in the Pod spec means a fresh claim is generated per Pod. A standalone ResourceClaim (like the first example) means the claim is reusable across pods and only released when explicitly deleted.

Run it:

kubectl apply -f smoke-test.yaml
kubectl -n gpu-test logs smoke-test -c cuda-app

You should see the nbody simulation running on GPU 0. The Pod's claim name is something like gpu-test/smoke-test-gpu-claim. Inspect it:

kubectl -n gpu-test get resourceclaim
kubectl -n gpu-test describe resourceclaim <claim-name>

Look at .status.allocation.devices — that's the actual GPU UUID the scheduler assigned.

Step 4: Migrate workloads one at a time¶

Pick the lowest-risk workload first. A dev cluster, a batch job, a CI runner — something where a 5-minute rollback is fine.

The mechanical conversion:

Before (device plugin):

spec:
  containers:
  - name: trainer
    resources:
      limits:
        nvidia.com/gpu: 4

After (DRA):

spec:
  resourceClaims:
  - name: training-gpus  # claim must exist in same namespace
  containers:
  - name: trainer
    resources:
      claims:
      - name: training-gpus
---
apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
  name: training-gpus
  namespace: ml-team
spec:
  devices:
    requests:
    - name: gpu
      deviceClassName: gpu.nvidia.com
      count: 4

The order of operations matters:

1. Create the ResourceClaim (or ResourceClaimTemplate).
2. Update the Pod spec (or Deployment template) to reference the claim.
3. Apply.
4. Verify the Pod schedules and the GPU shows up inside the container (nvidia-smi from inside).
5. Move to the next workload.

Don't try to migrate a Deployment by editing the manifest in place unless you have rollback at hand. Add the claim and the reference, deploy, then remove the nvidia.com/gpu: N line.

Step 5: Try a multi-GPU claim with shared mode¶

The device plugin had two ways to share a GPU: time-slicing (admit multiple pods to the same GPU via config) and MPS (true concurrency). DRA expresses sharing differently — through consumers lists:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: shared-gpu
spec:
  devices:
    requests:
    - name: shared-gpu
      deviceClassName: gpu.nvidia.com
      count: 1
  config:
  - requests:
    - name: shared-gpu
      capacity:
        requests: 4  # share 1 GPU across 4 consumer pods

Two pods reference the same claim template; the DRA driver exposes fractional GPU access to each. This is roughly equivalent to device-plugin time-slicing, but expressed in the API instead of in node-level config. The advantage: sharing is per-claim, not per-node. You can have one GPU shared across 4 inference pods and another GPU exclusive to a training job on the same node.

What you don't get from DRA sharing today: per-pod memory limits. The device plugin's time-slicing config can advertise a fraction of memory; DRA's sharing is currently memory-agnostic. If you need hard memory isolation between shared tenants, use MIG instead.

Step 6: Try MIG through DRA¶

MIG (Multi-Instance GPU) on the device plugin required a node-level config and a separate nvidia.com/mig-1g.5gb extended resource per profile. DRA expresses MIG as a DeviceClass selection:

apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
  name: mig-slice
spec:
  devices:
    requests:
    - name: mig
      deviceClassName: mig.nvidia.com
      config:
      - requests:
        - name: mig-profile
          selector:
            "nvidia.com/profile": "1g.10gb"  # H100 10GB MIG slice

The MIG kubelet plugin (part of NVIDIA's DRA driver) handles the underlying profile enablement. The Pod gets a CUDA-visible MIG instance corresponding to the requested profile. Same hardware, cleaner API — and you can mix MIG and non-MIG workloads on the same node without restarting anything.

GPU allocation in the DRA driver (gpuResourcesEnabled=true) is what unlocks MIG and other advanced allocation features. Enable it when you're ready, after all device-plugin workloads are migrated.

Step 7: Try Multi-Node NVLink (the headline feature)¶

This is the part the device plugin literally cannot do. NVIDIA's GB200 NVL72 racks need every GPU in a domain to be NVLink-reachable, with IMEX primitives (channels) coordinating cross-node GPU memory access. The DRA ComputeDomain resource expresses this:

apiVersion: resource.nvidia.com/v1alpha1
kind: ComputeDomain
metadata:
  name: gb200-training-domain
spec:
  numNodes: 2  # span across 2 nodes
  # Plus: channel configuration for IMEX
---
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: mnnvl-gpus
spec:
  devices:
    requests:
    - name: compute-domain
      deviceClassName: compute-domain.nvidia.com

When a Pod references this claim, the scheduler:

1. Finds a ComputeDomain with free capacity (or creates one).
2. Reserves the right number of GPUs across the right number of nodes.
3. Sets up IMEX channels between the nodes.
4. Attaches all GPUs to the Pod as one NVLink-coherent domain.

When the Pod terminates, the ComputeDomain tears down — IMEX channels are released, GPUs return to the pool. The ComputeDomain kubelet plugin is officially supported as of v0.4.1²; the GPU allocation side is still preview.

You need GB200 (or later NVLink-coherent) hardware to actually exercise this. On H100 or older, the compute-domain plugin will install but never match — your pods stay pending with no devices available until you remove the ComputeDomain claim.

Step 8: Decommission the device plugin¶

Once every workload in the cluster is on DRA:

# 1. Verify no Pod still uses nvidia.com/gpu
kubectl get pods -A -o json | \
  jq '.items[].spec.containers[].resources.limits
        | select(."nvidia.com/gpu" != null)' | head

# 2. Uninstall GPU Operator's device plugin component
# OR uninstall the device plugin DaemonSet directly
kubectl delete daemonset -n nvidia-device-plugin nvidia-device-plugin

# 3. If using GPU Operator, disable device plugin in values
helm upgrade gpu-operator nvidia/gpu-operator \
  --reuse-values \
  --set devicePlugin.enabled=false

# 4. Enable GPU allocation in DRA driver if you want full feature set
helm upgrade dra-driver-nvidia-gpu \
  oci://registry.k8s.io/dra-driver-nvidia/charts/dra-driver-nvidia-gpu \
  --reuse-values \
  --set gpuResourcesEnabled=true

Confirm nothing else references nvidia.com/gpu before you delete the DaemonSet. A leftover reference will manifest as 0/N Pods scheduled — easy to miss if you're not grepping for it.

When to skip DRA entirely¶

DRA is not a free upgrade. Reasons to stay on the device plugin:

• Cluster on K8s 1.31 or earlier — DRA is alpha/beta there. You can enable the feature gate, but you'll hit bugs that haven't been fixed yet because most production users are on 1.33+ or 1.34+¹.
• You only use whole GPUs, no MIG, no MNNVL, no sharing — the device plugin works fine for this. The migration cost (rewriting every workload's spec, training your team on claims) is not worth the cleanup.
• You depend on gpu-feature-discovery heavily and rely on the node labels it exports — GFD still works, but the labels become advisory once DRA is in play. The scheduler uses the claim, not the label.
• You have a custom scheduler or operator that reads nvidia.com/gpu extended resources directly — those will need rewriting.

For greenfield clusters on K8s 1.35+, skip the device plugin entirely. Install the DRA driver directly.

What I couldn't verify¶

• NVIDIA DRA driver GPU allocation GA timeline — v0.4.1 release notes label GPU features "for exploration and demonstration" with no published GA date². The gpu-kubelet-plugin is disabled by default. Treat all GPU-side claims as preview in mid-2026.
• MIG per-tenant memory isolation through DRA — the API supports per-claim MIG profile selection, but I couldn't confirm whether the DRA driver enforces per-instance memory caps the way MIG hardware does natively.
• DRA + Cluster Autoscaler integration — the CA ProvisioningRequest mechanism is supposed to work with DRA, but I couldn't find a published end-to-end walkthrough for GPU node provisioning through DRA + CA in mid-2026.
• DRA + Karpenter integration — Karpenter's NodePool CRD doesn't have first-party DRA awareness as of v1.13.0 (2026-06-10). If you use Karpenter for GPU autoscaling, the integration is custom or pending.
• Real-world adoption metrics — NVIDIA, Google Cloud (GKE has GA DRA support), and a handful of AI infra vendors are running it in production. Public deployment numbers (clusters, GPU scale-out) are not published.

Summary¶

• DRA is GA in Kubernetes 1.35. Use 1.32-1.34 only if you're testing.
• Install the NVIDIA DRA driver with gpuResourcesEnabled=false first. Enable the GPU allocation side only after you've migrated all workloads.
• Keep the device plugin running during the migration window. The two systems don't conflict at the API level if you're careful.
• Migrate one workload at a time — claim-based, then remove nvidia.com/gpu.
• ComputeDomain (MNNVL) is the headline feature and is officially supported. GPU allocation (MIG, sharing, time-slicing via API) is preview.
• Skip the device plugin entirely for greenfield clusters on K8s 1.35+. Migration cost only makes sense if you have an existing cluster.

The migration is real work — every workload manifest changes — but the API is what it should have been in 2018. The device plugin is still maintained; you don't have to move tomorrow. But if you're starting fresh or planning a K8s version jump, build the new cluster on DRA.

Questions or discussion? Connect on LinkedIn, X or reach out via email.

Discussion

Have thoughts on this post? Share them below — questions, corrections, or your own experience are all welcome.