Disaggregated LLM Inference on Kubernetes with LLM-D: Prefill/Decode Split, Expert Parallelism, and Real Autoscaling

LLM inference has two phases with different bottlenecks. Prefill is FLOPs-bound, decode is memory bandwidth-bound. Most production stacks shuffle both through the same GPU pool, which costs you time-to-first-token and tethers decode latency to whoever next requests a long prompt. LLM-D — the CNCF Sandbox project co-founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA — splits those phases onto separate pods, schedules expert parallelism for MoE models, and replaces CPU/GPU autoscaling with EPP-derived signals that actually predict load.

Two phases, one pool, lots of interference¶

Every token a chat model generates flows through two distinct phases:

• Prefill processes the entire prompt at once. The GPU is doing matrix multiplies — satellite FLOPs utilization. Latency here is the user's time-to-first-token.
• Decode generates one token at a time, autoregressively. Each step reads the prior KV cache, attends, appends, moves on. Memory bandwidth is the constraint. Latency here is time-per-output-token.

A standard ServingDeployment hands both phases to the same vLLM instance. The instance is fine at batching — until a 4,000-token prompt lands, eats the KV cache, and stalls 30 smaller requests waiting for decode. Disaggregation resolves this by carving the request: prefill runs on a specialized instance, the KV cache is shipped over RDMA to a decode instance, then the decode instance streams tokens back to the client.

LLM-D is the Kubernetes-native version of this pattern. The project hit CNCF Sandbox on 2026-03-24 and released v0.8.1 on 2026-06-26 (Apache-2.0, 3,875★).¹²

The architecture in three pieces¶

LLM-D's architecture has three components, and the boundaries between them are sharp.³

Variants inside an InferencePool are expressed as labels, not separate resources. The role label llm-d.ai/role takes values prefill, decode, or prefill-decode. The base kustomize layer also adds llm-d.ai/model, llm-d.ai/guide, llm-d.ai/accelerator-variant, llm-d.ai/accelerator-vendor, and llm-d.ai/engine-type, so the EPP can filter by model, hardware, and engine in policy plugins.⁴

The EPP component moved out of the Gateway API Inference Extension repo into llm-d/llm-d-router over v0.8. GAIE itself still hosts the InferencePool API spec and conformance tests.⁵ If you cite code, point to the new home.

Prefill/decode disaggregation in practice¶

The reference guide ships a working deployment: 8 × Prefill instances at TP=1 and 2 × Decode instances at TP=4, all serving openai/gpt-oss-120b.⁶ The Prefill Deployments look like this in practice.

- name: modelserver
  command: ["vllm", "serve"]
  args:
    - "openai/gpt-oss-120b"
    - "--tensor-parallel-size=1"
    - "--block-size=128"
    - "--kv-transfer-config"
    - '{"kv_connector":"NixlConnector", "kv_role":"kv_both"}'
    - "--no-disable-hybrid-kv-cache-manager"
  env:
    - name: VLLM_NIXL_SIDE_CHANNEL_HOST
      valueFrom: { fieldRef: { fieldPath: status.podIP } }
  resources:
    limits: { cpu: "8", memory: 16Gi, nvidia.com/gpu: "1" }

The KV cache transfer is NIXL (NVIDIA Inference Transfer Library), with UCX, UCCL, and libfabric as the underlying transports. NIXL is fast enough to run over RoCE or InfiniBand, with TCP fallback only for development. vLLM uses the NixlConnector with kv_both; SGLang uses a concurrent bootstrap protocol. A handshake between a new prefill/decode pair takes ~5 seconds once, then the connection is reused.⁷

The disaggregation decision isn't made by the model server; it's a policy in the EPP. The default EndpointPickerConfig for prefill/decode wires these plugins in order:⁸

apiVersion: llm-d.ai/v1alpha1
kind: EndpointPickerConfig
plugins:
  - type: disagg-headers-handler
  - type: always-disagg-pd-decider
  - type: disagg-profile-handler
    parameters: { deciderPluginName: always-disagg-pd-decider }
  - type: prefill-filter
  - type: decode-filter
  - type: approx-prefix-cache-producer
    parameters: { maxPrefixTokensToMatch: 131072 }
  - type: inflight-load-producer
  - type: prefix-cache-affinity-filter
    parameters: { peakPrefillThroughput: 33821 }
  - type: token-load-scorer
  - type: active-request-scorer
  - type: max-score-picker
schedulingProfiles:
  - name: prefill
    plugins: [prefill-filter, prefix-cache-affinity-filter,
              token-load-scorer, max-score-picker]
  - name: decode
    plugins: [decode-filter, active-request-scorer, max-score-picker]

The decider decides whether to disaggregate at all. If the user's prompt is already cached on a decode pod, prefill is wasted work. The prefix-cache-affinity-filter traces KV cache slots and pulls prompts to the decode pod that already has them. The two deployment shapes — prefill (TP=1) and decode (TP=4) — give the spec project the asymmetric memory profile it needs.

The claim numbers in the wild are serious. AWS measured up to 70% higher tokens/sec on GPT-OSS over standard vLLM on p6-b200 hardware.⁹ Oracle saw 10–30% on GPT-OSS-120B and Llama 3.3 70B on AMD MI300X. A Llama 3.1 70B deployment on 4× MI300X reportedly hit 3× throughput and 2× TTFT versus round-robin.¹⁰ Take partner numbers as partner numbers, but the order of magnitude is consistent.

Expert parallelism for MoE¶

Disaggregation also helps Mixture-of-Experts models. The DeepSeek-R1 / R2 / V4 lineage and Llama 4 use hundreds of experts per layer; only a few are active per token. The reference deployment in guides/wide-ep-lws/ is 32 × H200 or B200 GPUs running DeepSeek-R1-0528 with prefill DP=16 and decode DP=16 (16 replica pods in each role, each pod containing the full model with experts sharded across its GPUs via EP).¹¹

This is not "one pod per expert." Practitioners regularly confuse MoE expert parallelism with routing experts to pods. The actual layout is:

• Data Parallel across pods (16 replicas for each role).
• Expert Parallel within a pod (experts sharded across the GPUs in the worker set).
• LeaderWorkerSets (LWS) give the decode pod a single stable network identity while the workers scale horizontally.

The networking expectation is hard. NVIDIA DeepEP backends need all-to-all RDMA — a full NIC-to-NIC mesh between pods. Rail-only connectivity fails. Intel XPU uses XCCL with allgather_reducescatter instead but still requires the full mesh. LLM-D ships a custom vLLM image for this guide because v0.23.0–v0.24.0 had a DP-supervisor bug and NVSHMEM misbehaves on RoCE under DeepEP HT.

The EPP gained DP-aware scheduling in v0.8 (status: Graduating). For wide-EP, the LeaderWorkerSet pods wear the llm-d.ai/role=prefill or decode label set, and the EPP routes based on DP rank.

Autoscaling that doesn't lie¶

This is the part most teams get wrong. CPU utilization in the LLM is decoupled from real workload. GPU utilization is pegged at 100% during active batching regardless of amount of rear-arriving work. Both are lagging indicators. LLM-D rejects both as primary signals and ships two complementary paths.¹²

Path A: KEDA + EPP metrics¶

KEDA polls EPP-emitted flow-control signals and creates/owns the HPA. The signals are real:

• EPP Flow Control queue depth
• Running request count per pod

This is the obvious choice for homogeneous hardware and independently scaled pools. It supports scale-to-zero without the HPAScaleToZero feature gate.

Path B: HPA + WVA (Workload Variant Autoscaler)¶

WVA consumes supply-side signals and is meant for mixed environments. The WVA repository is Apache-2.0, and the CRDs moved to the llm-d.ai API group in v0.8.0.¹²¹³

What WVA sees that KEDA doesn't:

• KV cache utilization
• Model-server queue depth
• Performance budgets / SLO targets
• Gateway queue size

WVA emits a wva_desired_replicas metric. HPA (or KEDA) consumes it via the external metrics API. Because WVA has a global view across heterogeneous hardware, multiple variants, and multiple InferencePools, it can gracefully degrade when accelerator inventory is short — it maximizes SLO attainment rather than maximizing replicas.

Pick a path. KEDA is faster to deploy. WVA is the answer when you have GPU heterogeneity or want SLO-driven scaling.

The Gateway API Inference Extension¶

The InferencePool CRD is the wiring layer. It comes from kubernetes-sigs/gateway-api-inference-extension (currently v1.5.0, released 2026-04-19).¹⁴

apiVersion: inference.networking.k8s.io/v1
kind: InferencePool
metadata: { name: gpt-oss, namespace: llm-d }
spec:
  selector:
    matchLabels: { app: baseline }
  targetPorts:
    - { number: 8000 }
  endpointPickerRef:
    name: gpt-oss-epp
    kind: Service
    port: { number: 9002 }
    failureMode: FailOpen

InferencePool is namespaced. One model deployment maps to one pool. The pool owns the EPP service; the EPP owns the plugin config. The Gateway controller wires the proxy's ext_proc filter to the EPP. Supported gateway providers: Envoy Gateway, GKE Gateway, Istio, and kgateway (agentgateway).¹⁵

The only request_body_mode and response_body_mode the EPP supports is FULL_DUPLEX_STREAMED. Get that wrong and Envoy will buffer the request body and your streaming latency goes through the roof.

Install¶

LLM-D is helm + kustomize, not an Operator (yet). The PD-disaggregation guide produces a working install in three steps.⁶

# Pin versions
export GAIE_VERSION=v1.5.0
export ROUTER_CHART_VERSION=v0.4.0
export GUIDE_NAME=pd-disaggregation
export NAMESPACE=llm-d

# 1. Install GAIE CRDs
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/${GAIE_VERSION}/v1-manifests.yaml

# 2. Install llm-d Router (Envoy + EPP)
helm install ${GUIDE_NAME} \
  oci://ghcr.io/llm-d/charts/llm-d-router \
  -f https://raw.githubusercontent.com/llm-d/llm-d/main/guides/recipes/router/base.values.yaml \
  -f https://raw.githubusercontent.com/llm-d/llm-d/main/guides/${GUIDE_NAME}/router/${GUIDE_NAME}.values.yaml \
  -n ${NAMESPACE} --version ${ROUTER_CHART_VERSION}

# 3. Apply model-server manifests
kubectl apply -k https://github.com/llm-d/llm-d/guides/${GUIDE_NAME}/modelserver/gpu/vllm/base

Wait for the model-server pods to finish loading shards, then test:

kubectl port-forward svc/${GUIDE_NAME}-decode 8000:8000 -n ${NAMESPACE} &
curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"openai/gpt-oss-120b","messages":[{"role":"user","content":"hello"}],"max_tokens":32}'

vLLM compatibility¶

LLM-D v0.8.1 ships against vllm/vllm-openai:v0.23.0 upstream. Older releases used a custom llm-d-cuda image but v0.8.0 dropped that for the upstream vLLM image and a set of supported accelerators (CUDA, ROCm, XPU, HPU, AWS).²

If you want a different vLLM, the kv transfer config is the constraint. The NixlConnector is the working stack; the Mooncake connector was added in v0.8.0 (also supported now but more typically used for the inference simulator).

A practical first pass¶

• Start with a single-Pool deployment (no P/D) to validate the gateway integration.
• Move to P/D with openai/gpt-oss-120b on the reference 8 prefill / 2 decode shape.
• Add prefix-cache-aware routing (prefix-cache-affinity-filter, approx-prefix-cache-producer) and watch TTFT drop.
• Switch to KEDA + EPP metrics for HPA. Skip the WVA signal set until you have heterogeneous hardware.
• Upgrade to vLLM v0.23.0 if you have the headroom — newer LLM-D releases assume it.
• For MoE, validate your network fabric actually supports all-to-all RDMA before commiting.
• Wire OTel traces into the EPP ext_proc path so disaggregation events are visible in Tempo/Honeycomb.

Summary¶

• LLM-D disaggregates prefill and decode onto separate vLLM pods with prefix-cache-aware routing and NIXL-based KV transfer.
• The EPP replaced centralized routing with a per-pool plugin pipeline that decides when to disaggregate, what to filter, and how to score.
• Wide-EP for MoE is data-parallel across pods, expert-parallel within pods, with LWS giving the cluster a single identity.
• Autoscaling is via KEDA + EPP metrics or HPA + WVA — never CPU or GPU utilization.
• The InferencePool CRD comes from GAIE; the EPP source has moved to llm-d/llm-d-router.
• v0.8.1 (2026-06-26) is the current release; vLLM v0.23.0 is the default image.

The LLM serving stack is moving from "one big model on many GPUs" to "specialized phases on purpose-built pods." LLM-D is the Kubernetes-native expression of that shift.

Questions or discussion? Connect on LinkedIn, X, or email.

Discussion

Have thoughts on this post? Share them below — questions, corrections, or your own experience are all welcome.