Serving an LLM on Kubernetes in 2026: the operations checklist nobody gave you¶

You can helm install vllm in an afternoon. You cannot tell me your p99 TTFT at 70% GPU utilization — and that's the gap this post is about. The "deploy vLLM on Kubernetes" tutorial is saturated. The decision tree underneath it isn't written anywhere, so most clusters are still pinned to whatever they stood up 12 months ago.

The engine layer moved while you weren't looking: Tiny-vLLM (May 29), KVarN KV-cache quantization (June 4), Kvcached for elastic KV cache, Expanse reclaiming idle GPU, and vLLM's wide expert-parallel path pushing ~2.2k tok/s on H200s. (All five are bleeding-edge — verify the names, dates, and numbers before you publish or pin them.) Meanwhile your Helm chart hasn't changed since last summer.

This post is the engine-and-hardware layer: which engine, which quantization, how to autoscale GPUs that don't behave like CPUs, and when to admit you should just call an API. For the scaling-patterns deep dive, see GPU Autoscaling is Broken; to size the fleet before you buy, start with the GPU memory capacity-planning guide.

Is this for you?¶

You're running — or about to run — vLLM, TensorRT-LLM, or SGLang on your own GPUs, and you have more confidence in your Helm values than in your p99. If your answer to "what's your TTFT under load?" is a shrug, keep reading.

If you're calling a hosted API and happy, skip to the last section — it might save you a cluster.

The one decision that dominates everything: which engine¶

Engine choice is upstream of every ops decision you'll make. Pick wrong and no amount of HPA tuning saves you. Here's the 2026 matrix.

My default recommendation: start on vLLM. It's the path of least resistance, runs everywhere, and the V1 engine closed most of the throughput gap. You graduate off it only for a specific reason.

Three reasons to graduate:

• You're NVIDIA-only, traffic is steady, and you've squeezed vLLM dry → TensorRT-LLM. It wins on peak tok/s and tail latency when tuned, but you pay with a per-model/per-GPU compile step that wrecks your rollout story. Don't take this on unless throughput is literally the product.
• You're agentic, multi-turn, or RAG-heavy with massive shared prefixes → SGLang. RadixAttention reuses KV across requests that share a prefix, which is exactly the agent/tool-loop pattern. On those workloads it can beat vLLM without you touching a flag.
• You're CPU-bound, on the edge, or serving a handful of QPS → llama.cpp. GGUF on a CPU node is cheaper and simpler than a GPU you'll keep 5% busy. No shame in it.

                         ┌─ NVIDIA-only + steady high volume + tuned-out vLLM? ── TensorRT-LLM
 Need GPU serving? ──┬── ┼─ Agentic / multi-turn / big shared prefixes? ───────── SGLang
                     │   └─ Otherwise ──────────────────────────────────────────── vLLM (V1)
                     └── Low QPS / edge / CPU-only? ──────────────────────────────── llama.cpp

Quantization and KV-cache tricks that actually move the needle¶

Most "optimization" advice is noise. Here's the short list that changes your tok/s or your GPU bill, ranked by payoff.

FP8 weights + FP8 KV cache (Hopper/Blackwell). This is the single biggest lever in 2026. Near-lossless quality, ~2x effective memory, and the math runs on native tensor cores instead of being emulated.

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --quantization fp8 \
  --kv-cache-dtype fp8 \
  --max-model-len 32768

That FP8 KV cache is the underrated half. The KV cache — not the weights — is what caps your concurrency at long context. Halving its footprint roughly doubles how many concurrent long-context requests fit before you start preempting.

AWQ / GPTQ (4-bit weights) when you're memory-bound on older GPUs. On Ampere (A100) without FP8, 4-bit AWQ is how you fit a 70B on a single 80GB card with room for KV. Quality cost is small; throughput on decode improves because you're moving fewer bytes.

Automatic prefix caching — turn it on, it's free throughput. If your prompts share a system preamble or a RAG document, vLLM caches the KV for that prefix across requests. One flag:

--enable-prefix-caching

For agent workloads with a 2,000-token system prompt repeated on every call, this alone can cut TTFT dramatically because prefill skips the cached span.

The 2026 research wave — KVarN, Kvcached, Expanse — is about elastic KV. The common thread: stop treating KV cache as a fixed pre-allocated slab. KVarN quantizes it further; Kvcached makes it elastic so idle capacity is reclaimable; Expanse reclaims idle GPU outright. Promising, fast-moving, not yet boring-enough-for-production — pilot before you bet a cluster on them.

One layer up: prompt caching at the API boundary. If part of your traffic still hits a hosted model (Bedrock, Anthropic, others), their prompt caching bills the cached prefix at a steep discount. Same principle as prefix caching, different wallet — don't pay full price for a system prompt you send a million times.

Autoscaling that respects GPU topology¶

Here's where most teams ship a broken setup: they scale LLM pods on CPU or memory, which are uncorrelated with whether the model is actually drowning. GPUs aren't fungible CPU slices. Scale on the signals that matter.

Scale on queue depth and KV pressure, not CPU. The two real signals are vllm:num_requests_waiting (requests queued, not yet running) and KV-cache utilization (vllm:gpu_cache_usage_perc). KEDA's Prometheus scaler reads them directly:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-scaler
spec:
  scaleTargetRef:
    name: vllm
  minReplicaCount: 1            # never cold — keep one warm
  maxReplicaCount: 8
  cooldownPeriod: 300           # don't thrash a 4-min-to-load model
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        metricName: vllm_requests_waiting
        query: sum(vllm:num_requests_waiting{app="vllm"})
        threshold: "10"          # >10 queued per replica → scale out
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        metricName: kv_cache_usage
        query: avg(vllm:gpu_cache_usage_perc{app="vllm"})
        threshold: "0.85"        # cache >85% full → scale out

Node provisioning is a separate problem — that's Karpenter's job, not KEDA's. KEDA decides you need another pod; Karpenter finds (or boots) a node with the right GPU. Constrain it to the SKUs you actually validated, or it'll happily place a tensor-parallel shard on a node with no NVLink and tank your throughput:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-h100
spec:
  template:
    spec:
      requirements:
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["p5.48xlarge"]      # 8x H100, NVLink — validated
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
  disruption:
    consolidationPolicy: WhenEmpty      # don't consolidate mid-generation
    consolidateAfter: 600s

Topology is the trap. If you run tensor parallelism across GPUs, those GPUs must share NVLink. Scaling a TP=4 model onto four PCIe-only GPUs doesn't just run slower — interconnect becomes the bottleneck and you can lose more than half your throughput. Pin the instance type. Don't let the scheduler improvise.

Why it bites: tensor parallelism does an all-reduce across GPUs after every layer, on every token — thousands of tiny, latency-sensitive collectives per second. NVLink moves them at ~900 GB/s; PCIe at ~64 GB/s. On the slow link the GPUs sit idle waiting on the network, and your expensive compute is bottlenecked by a bus.

Two rules fall out: keep a TP group inside one node (NVLink doesn't cross nodes — go pipeline-parallel or InfiniBand instead), and verify the wiring before you trust a node — PIX/PHB/SYS in the matrix below means PCIe, not NVLink:

nvidia-smi topo -m   # NV# = NVLink; PIX/PHB/SYS = PCIe/host bridge (no NVLink)

DRA is how this gets cleaner. Dynamic Resource Allocation (GA-track in recent Kubernetes) lets you request GPUs by capability — "give me 2 GPUs with NVLink and ≥80GB" — instead of the blunt nvidia.com/gpu: 2 count, which can't express topology at all. If you're standing up a new cluster in 2026, build on DRA rather than the legacy device plugin.

Cold-start math, multiplexing, and the honest exit¶

Cold-start is your real scaling latency¶

When KEDA says "scale out," the clock that matters isn't pod scheduling — it's time to first served token on the new replica. Budget every stage:

The lesson: you cannot autoscale your way out of a spike on a 5-minute cold start. Keep a warm floor (minReplicaCount: 1+), scale early on queue depth, and stage weights on local NVMe so load time isn't a registry round-trip.

Multi-model multiplexing on one node¶

If you've got several small models and low per-model utilization, packing them onto one GPU beats stranding a card per model:

# Two models sharing one H100, ~40GB each
vllm serve model-a --port 8000 --gpu-memory-utilization 0.45
vllm serve model-b --port 8001 --gpu-memory-utilization 0.45

The catch: KV cache is per-process, so two models on one GPU serve roughly 70% of what two dedicated cards would, not 100%. The math only works at low average utilization. At high load, give each model its own GPU.

When to just call an API¶

The unglamorous truth: a GPU bills you whether or not it's busy, and an API only bills you when you call it. That asymmetry decides self-host vs. API more often than any benchmark.

The break-even, concretely. An H100 runs roughly $3/hr on-demand → ~$2,160/month, fixed. Say a hosted equivalent charges ~$0.60 per 1M output tokens. To beat the API, your own GPU has to generate:

$3/hr ÷ $0.60 per 1M tokens = 5,000,000 output tokens/hr = ~1,400 tok/s, sustained 24/7

A single H100 can do ~1,400 tok/s aggregate on a quantized 70B — but only at high batch. If you can't keep it busy at ~50%+ of that around the clock, the API is cheaper and you've skipped the on-call rotation. Self-hosting wins on three axes the spreadsheet hides: steady high volume, data residency, and latency control. If none of those is your reason, you're paying $2K/month for a server-shaped hobby.

Run the numbers honestly before you provision a cluster. The best LLM-on-Kubernetes decision is sometimes not to.

Summary¶

The decision tree, top to bottom:

• Engine — default vLLM (V1); TensorRT-LLM only for tuned NVIDIA high-volume; SGLang for agentic/prefix-heavy; llama.cpp for edge/CPU/low-QPS
• Quantization — FP8 weights and FP8 KV cache on Hopper/Blackwell; AWQ 4-bit when memory-bound on Ampere
• Prefix caching — --enable-prefix-caching on; prompt caching at any API boundary
• Autoscale on num_requests_waiting + KV utilization, never CPU
• Pin GPU topology — Karpenter to validated NVLink SKUs; build new clusters on DRA
• Budget cold start at 2–6 min — keep a warm floor, scale early, stage weights on local NVMe
• Multiplex small models at low utilization only (~70% efficiency)
• Run the API break-even (~1,400 tok/s sustained per H100) before you self-host

Questions or discussion? Connect on LinkedIn, X or reach out via email.

Discussion

Have thoughts on this post? Share them below — questions, corrections, or your own experience are all welcome.