Build Your Own Token-as-a-Service: A Self-Hosted OpenAI-Compatible AI Gateway on Kubernetes¶

Public LLM APIs bill you per token and leak your prompts off-prem. The fix isn't "run Ollama on a box" — it's a centralized, OpenAI-compatible inference platform your whole org consumes like a utility: one API key, token-based quotas, models that stay in your datacenter. Call it token-as-a-service. This post is the architecture — model tiering, GPU math, MIG partitioning, vLLM tuning, and KV-cache-aware routing — generic enough to run on any Kubernetes cluster with NVIDIA GPUs.

The reference target: ~1,750 concurrent users during business hours, split across interactive chat, RAG, long-document synthesis, and agentic coding — served from a fixed pool of GPUs without overflowing into cloud spend. Everything below is sized against that.

The shape of the platform¶

Three layers, each independently scalable:

Keep the gateway off the GPUs. The gateway is CPU- and I/O-bound — auth, JSON, rate-limit counters. Pin it to standard CPU nodes so every gram of GPU memory goes to active inference. It also makes the gateway trivially HA: CPU pods migrate during maintenance, GPU pods don't have to.

client / IDE / agent
        │  OpenAI schema + API key
        ▼
┌──────────────────────┐   CPU-only node pool
│  Envoy Gateway (x2)   │  auth · rate limit · audit
│  + inference routing  │  weighted routes per model
└─────────┬────────────┘
          │ weighted-cluster route → Endpoint Picker (EPP)
   ┌──────┴───────┐
   ▼              ▼
 prod cluster   staging cluster      bare-metal GPU node pools
 (vLLM pools)   (vLLM pools)         planner · builder · reviewer · support

Model tiering: stop using one big model for everything¶

A single monolithic model is the wrong default. Different model families have different strengths, and an agentic coding flow has at least three distinct jobs — plan, build, review. Routing each through a model suited to it produces higher-quality output and catches mistakes a single model misses.

Every model below is open-weight and on Hugging Face — swap in whatever your evals favor, but these are concrete, runnable starting points:

Strong open alternatives if you want a second opinion in your evals: Qwen/Qwen3-30B-A3B⁴ (MoE builder/planner), Qwen/Qwen3-Coder-30B-A3B-Instruct⁵ (coding-tuned builder), and google/gemma-3-27b-it⁶ (dense reviewer). Validate the actual checkpoint against your own benchmarks before committing GPUs.

Plus the support tier — small open models that do the unglamorous work, each well under 8B parameters:

These are tiny. Handing each a full 96 GB GPU wastes 80–90% of the card. That's exactly what MIG fixes — more below.

GPU capacity is KV-cache math, not parameter count¶

Model weights are the small part of GPU memory at scale. The real consumer is the KV cache — and it grows with concurrent sessions and context length, not model size.

Each concurrent request reserves memory roughly proportional to its context (input + output). On a 96 GB card running FP8 KV cache, rough per-session costs:

That converts directly to a replica count. Take the available KV budget per replica, divide by per-session cost, divide your user target by that:

planner   1,500 chat users × 0.6 GB ÷ ~200 users/replica  →  ≥ 8 replicas
builder     200 RAG users × 1.5 GB ÷ ~110 users/replica   →  ≥ 2 replicas
reviewer     50 deep users × 8 GB  = ~400 GB aggregate KV  →  ≥ 2 replicas

Pick GPUs with enough HBM to keep peak KV cache fully resident. Long-context workloads (256K-capable models, 100K reviewer inputs) generate KV loads that overflow to host memory on small cards — and the moment KV spills, inter-token latency gets unstable. A 96 GB card leaves 64–88 GB free for KV after weights and gives you headroom.

Tensor parallelism shards weights across GPUs and frees more room for KV:

If your GPUs lack NVLink, all tensor-parallel traffic crosses PCIe Gen5 inside the node. It works, but keep TP groups within a single physical node — don't shard a model across the network.

MIG: turn one stranded GPU into four productive ones¶

Small support models on full GPUs is a waste. Multi-Instance GPU (MIG) carves a physical card into hardware-isolated slices, so embeddings, reranking, and guardrails each get a right-sized partition instead of a whole card.

This section is the why for a TaaS platform. For installing the GPU Operator from scratch see GPUs on Kubernetes, and for the full MIG + time-slicing values.yaml applied through GitOps see Stacking MIG + time-slicing.

The strategy that makes this work is mixed mode — the only MIG strategy that allows different layouts on a single node, so large-model passthrough GPUs and sliced small-model GPUs coexist.

Example layout per GPU node (8 GPUs each):

• First 2 GPUs → MIG-enabled, four 1g.24gb slices each = 8 slices for support models.
• Remaining 6 GPUs → full passthrough for the large model pools.

Drive it with node labels via the NVIDIA GPU Operator:

# Slice the MIG-enabled nodes
kubectl label node <node> nvidia.com/mig.config=mig-mixed-config --overwrite

# Keep the rest as full GPUs
kubectl label node <node> nvidia.com/mig.config=all-disabled --overwrite

Three structural wins:

• Density. Each MIG GPU yields 4 isolated slices — a clean 4:1 gain on otherwise-stranded capacity.
• Isolation. Hardware partitioning means a request burst on one endpoint can't monopolize compute or memory of a neighbor on the same card.
• Failure domains. Spreading MIG-enabled GPUs across multiple nodes distributes support-model capacity, so a single GPU or node loss doesn't take out a whole tier. That preserves N+1 for failover and canary releases.

The 1g.24gb profile cleanly matches the embedding/rerank/guardrail footprint. Bigger profiles (2g.48gb, 4g.96gb) exist if your support tier grows — match the slice to the model, don't over-allocate.

The hard limit: one model can't span MIG slices¶

This is the constraint that decides the whole layout, so state it plainly: a single inference process can only use one MIG slice. It's by design, not a bug or a missing config.

MIG instances are hardware-isolated — dedicated SMs, memory, and cache paths. NVIDIA deliberately blocks the cross-instance communication that multi-GPU inference needs:

• No NVLink, no peer-to-peer, no CUDA IPC between slices. Tensor parallelism relies on NCCL collectives that require P2P access between devices. MIG forbids it.
• So a MIG slice is always TP=1. You can attach multiple MIG devices to one pod, but the model server still can't fuse them into one tensor-parallel group — they sit there as separate, unusable-together devices.

One MIG slice = one model replica, period. If a model doesn't fit in your card's biggest slice, you're off MIG and onto full GPUs.

Deciding MIG vs. full GPU¶

That limitation turns into a simple decision tree:

Does the model fit in a single MIG slice (≤ largest profile, e.g. 4g.96gb)?
├─ Yes → does it need TP > 1 for throughput/latency?
│        ├─ No  → MIG slice, TP=1        ← support tier lives here
│        └─ Yes → full GPUs, TP=2/4      ← MIG can't shard, so don't
└─ No  → full GPUs, TP=2/4               ← too big for any slice

The trade-off in one line: MIG buys you density and isolation for small models, and costs you tensor parallelism. The moment a model needs to shard across GPUs, MIG is the wrong tool — give it whole cards.

Time-slicing: pack several models onto one slice¶

MIG gives you N isolated slices. But a 1g.24gb slice running one idle reranker still wastes most of its duty cycle. Time-slicing oversubscribes a slice so multiple model pods share it — the GPU context-switches between them in round-robin.

It stacks on MIG. You partition with MIG, then advertise each slice R times through the NVIDIA device plugin's timeSlicing config:

# device-plugin config — oversubscribe each MIG slice 3×
sharing:
  timeSlicing:
    resources:
    - name: nvidia.com/mig-1g.24gb
      replicas: 3

# Point the MIG-enabled nodes at the time-sliced profile
kubectl label node <node> nvidia.com/device-plugin.config=mig-timeslice --overwrite

Now kubectl describe node shows 3 schedulable mig-1g.24gb units per physical slice — so the 8 slices on a node become 24 schedulable units. The density multiplies: MIG slices × time-slice replicas = total units.

Full working values for stacking MIG + time-slicing through the GPU Operator are in Stacking MIG and Time-Slicing on One GPU Operator values.yaml.

The catch: time-slicing isolates nothing¶

MIG and time-slicing are opposites on the one axis that matters — isolation:

The LLM-specific landmine: time-slicing does not divide memory, so every vLLM replica on the slice sees the full 24 GB and, left at the default --gpu-memory-utilization 0.90, each tries to grab almost all of it. Three replicas → instant OOM. You must hand-cap each replica so the sum fits:

# 3 vLLM pods sharing one 24 GB slice → ~0.30 each, not 0.90
--gpu-memory-utilization 0.28

That tiny per-replica budget is why time-slicing only makes sense for the small, idle support tier — a reranker that fires for 20 ms per query, not a planner streaming tokens for 30 seconds.

Deciding: MIG, time-slicing, or both¶

Need hardware isolation / predictable latency?
├─ Yes → MIG only (no time-slicing). Production support models.
└─ No  → are the models small AND mostly idle/bursty?
         ├─ Yes → MIG + time-slicing. Pack dev/test or low-traffic models.
         └─ No  → full GPU. Throughput-bound or latency-sensitive work.

The rule that ties it together: MIG splits the silicon, time-slicing oversubscribes the splits — and the safe place to oversubscribe is exactly where isolation matters least. Production tenants get whole slices; dev and idle models share them.

vLLM engine tuning that actually moves the needle¶

Catalog defaults get you running; they don't get you optimal. The settings below are where throughput and latency are won. Per inference pool:

If you turn on KV offloading, raise the engine's CPU and RAM limits to match — e.g. 24 vCPU and enough host RAM to hold the offload buffer (a 200 GiB buffer wants ~384 GiB on the node). Offloading to DRAM that isn't there does nothing.

Treat host-DRAM KV offloading as upside, not baseline. Size your capacity for worst-case all-resident KV, then let offloading buy you extra concurrency and burst tolerance on top.

KV-cache-aware routing: don't recompute the conversation¶

Round-robin load balancing quietly destroys multi-turn performance. Every time a follow-up message lands on a different GPU, that GPU recomputes the entire conversation history from scratch — Time-to-First-Token climbs with every turn.

The fix is the Kubernetes Gateway API Inference Extension — an InferencePool plus an Endpoint Picker (EPP) that schedules requests based on backend state, not blind rotation. Envoy Gateway applies a weighted-cluster route per model, then fans out to the EPP, which picks the optimal model-server pod.

The EPP optimizes two things at once:

The architectural trick is decoupling ingress routing from backend scheduling: the gateway weights traffic across environments; the EPP picks the replica inside the target pool. Requires a minimum of 2 replicas per pool — for HA and so prefix-aware routing has somewhere to route.

Asymmetric prod/staging weights¶

Run staging as live failover capacity, but weight traffic by physical capacity, not 50/50:

Token-based rate limiting (RPM lies)¶

Rate-limit on tokens per minute, not requests — one request might burn 10 tokens, another 10,000. Requests-per-minute tells you nothing about GPU pressure.

Combine two scopes, evaluated independently; a request is rejected if it trips either:

• Global TPM — pool-wide ceiling shared across all keys (use a shared store like Redis across gateway replicas). The ultimate hardware safeguard: even a high-override key gets throttled once aggregate traffic hits the pool ceiling.
• Per-key TPM — individual buckets per API key.

Example ceilings by pool — note the Planner carries the lowest limits (accuracy over concurrency) and the MoE Builder the highest (cheap per-token decode):

No native tier concept? Encode it in the key name. A <tier>-<consumer>-<env> convention — enterprise-orderflow-prod, standard-marketing-bot-prod — turns a wall of opaque keys into a greppable, self-documenting policy surface. The prefix is the contract.

Governing agent tools with a single MCP endpoint¶

Agentic workflows want to call external systems — databases, metrics, ticketing. Don't let autonomous agents connect to internal systems directly. Put the gateway in the path as an interception proxy using the Model Context Protocol (MCP).

One endpoint gives you a single pane for monitoring, auditing, and authorization. The gateway intercepts every tool call from IDEs or agent orchestrators, verifies the key's permissions, and proxies execution down to the right cluster — so multi-turn reasoning chains are governed before they touch corporate data.

Infrastructure notes that bite if you skip them¶

• Separate the databases from the inference platform. Run PostgreSQL on its own VMs/instances with independent BCDR. Don't co-locate app state with GPU workloads.
• Mixed VM + bare-metal Kubernetes. Virtualized control plane and CPU workers, bare-metal GPU node pools joined as a separate pool. GPU nodes carry a taint so stray workloads can't land on them — add explicit tolerations for the inference pods only.
• Observability the platform won't give you for free. Add a vLLM service monitor and dashboards for: KV-cache metrics and query stats, plus Envoy overview / upstream / downstream. TTFT, queue wait, and token-generation rate are your real SLO signals — not CPU%.

Summary¶

The reusable pattern, end to end:

• Gateway on CPU nodes, inference on GPU nodes. Never burn GPU memory on auth and JSON.
• Tier your models by job — planner / builder / reviewer + small support models — instead of one monolith.
• Size GPUs by KV-cache math: per-session GB × user target ÷ replica capacity = replica count. Keep peak KV resident in HBM.
• MIG mixed-mode to slice small-model GPUs 4:1 while large models keep full cards. One slice = one model (TP=1) — span more than that and you're on full GPUs.
• Time-slice only the idle support tier to oversubscribe slices (slices × replicas units) — and cap each vLLM's --gpu-memory-utilization so they fit shared memory. Never time-slice latency-sensitive prod.
• Tune vLLM: fp8 KV cache, right-sized --max-model-len, prefix caching on, offload cold KV to DRAM as upside.
• Route with the Gateway API Inference Extension + EPP — prefix-aware and load-aware, not round-robin. Min 2 replicas/pool.
• Rate-limit on tokens (TPM), global + per-key, with tier baked into the key name.
• Front all agent tools with one governed MCP endpoint.

The one-line mental model: a self-hosted token-as-a-service platform is just the right model, on the right slice of GPU, reached by the route that already holds your KV cache — wrapped in token quotas so no single consumer can starve the rest.

Discussion

Have thoughts on this post? Share them below — questions, corrections, or your own experience are all welcome.