Skip to content

Kubernetes

MCP Servers in Production: Patterns for Connecting AI Agents to Kubernetes and Cloud APIs

The first wave of AI agents talked to infrastructure through bespoke wrappers: a Python script that called kubectl, a Lambda that polled CloudWatch, a chat plugin that hard-coded an OpenAPI spec. That worked for demos. It collapsed the moment an agent needed to talk to two systems at once, or get rotated credentials, or appear in a SOC 2 audit.

Model Context Protocol (MCP) is the answer that won, and by 2026 it has the ecosystem to prove it. The Python SDK alone shipped v2.2.0 on 2026-09-07.1 The spec was last revised 2025-06-18 and is now stable enough that AWS Labs, Microsoft, Google, Cloudflare, and Alibaba all ship vendor MCP servers against it.2 OWASP published an MCP-specific Top 10 in 2025 because the attack surface is real enough to warrant one.3

This post is the field guide I wish someone had handed me twelve months ago. It covers the transport choices, the Kubernetes deployment patterns, the cloud-vendor landscape, authentication, security, multi-server gateways, observability, and the operational checklist that turns a working prototype into a production control plane.

Self-Host an AI Gateway on Kubernetes: Routing, Caching, and Cost Control with Envoy

One LLM provider is a weekend project. Five providers, two self-hosted inference clusters, three regions, and a per-team budget is a platform. The path between them is an AI gateway — and Envoy is the only data plane with a first-class AI filter shipped upstream. Run it on Kubernetes the way you already run any other gateway, and you get a single control plane for every model call in the company.

Disaggregated LLM Inference on Kubernetes with LLM-D: Prefill/Decode Split, Expert Parallelism, and Real Autoscaling

LLM inference has two phases with different bottlenecks. Prefill is FLOPs-bound, decode is memory bandwidth-bound. Most production stacks shuffle both through the same GPU pool, which costs you time-to-first-token and tethers decode latency to whoever next requests a long prompt. LLM-D — the CNCF Sandbox project co-founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA — splits those phases onto separate pods, schedules expert parallelism for MoE models, and replaces CPU/GPU autoscaling with EPP-derived signals that actually predict load.

GPUs on Kubernetes: From Bare Metal to Schedulable in One Operator

A fresh Kubernetes cluster has no idea your nodes have GPUs. kubectl describe node shows CPU, memory, and pods — nothing else. To make a pod request a GPU you need a driver, a container runtime hook, and a device plugin advertising the hardware to the scheduler, all version-matched across every GPU node. Do it by hand and you'll re-do it on every kernel bump. This post wires it up the way you actually want — one operator — and works on any Kubernetes, not a specific vendor's distro.

Stacking MIG and Time-Slicing on One GPU Operator values.yaml

MIG carves a GPU into hardware-isolated slices. Time-slicing oversubscribes each slice so more pods can share it. Stack them and one physical GPU advertises far more schedulable units than it has silicon — useful when you have more workloads than GPUs and most of them sit idle. Here's the exact values.yaml, wired into the kommander-applications GPU Operator 26.3.0 app1, and the labels that switch a node between layouts.

Build Your Own Token-as-a-Service: A Self-Hosted OpenAI-Compatible AI Gateway on Kubernetes

Public LLM APIs bill you per token and leak your prompts off-prem. The fix isn't "run Ollama on a box" — it's a centralized, OpenAI-compatible inference platform your whole org consumes like a utility: one API key, token-based quotas, models that stay in your datacenter. Call it token-as-a-service. This post is the architecture — model tiering, GPU math, MIG partitioning, vLLM tuning, and KV-cache-aware routing — generic enough to run on any Kubernetes cluster with NVIDIA GPUs.

GPU DRA on Kubernetes: migrating off the NVIDIA device plugin, end-to-end

Dynamic Resource Allocation went GA in Kubernetes 1.351, and NVIDIA's DRA driver for GPUs shipped v0.4.1 on 2026-06-302. The nvidia.com/gpu extended resource model — the one the device plugin has used since 2017 — now has a sanctioned successor. The migration is non-trivial because the resource model is completely different (claim-based, not extended-resource-based), but the payoff is real: structured GPU allocation, native NVLink topology awareness, and a single API for MIG, MPS, time-slicing, and Multi-Node NVLink.

This post is the migration path: install the driver, write the first claim, swap one workload at a time, and run device plugin + DRA side-by-side until you're done.

Cilium and eBPF: the Networking Layer Under Your Cluster

Your cluster routes every packet through iptables, and at a few thousand Services that chain becomes a linear scan the kernel walks per connection. Cilium throws that out and runs networking, policy, and observability as eBPF programs in the kernel instead — no sidecars, no iptables, no agent in the data path. This post walks the whole stack: how a packet even finds another pod, installing Cilium as a kube-proxy replacement, watching real traffic with Hubble, locking the cluster down with identity-based policy, stretching it across clusters with Cluster Mesh, and dropping the service-mesh sidecars — all copy-paste.

Centralize Log Solution with the Elastic Stack

Use case: ship Kubernetes ingress logs off the cluster into a self-managed Elastic Stack on Ubuntu servers, parse them, and keep them searchable for 90 days without blowing the disk budget. The Elastic Stack handles this cleanly, and the 2026 build leans on four current features that older tutorials skip: LogsDB, ILM, data streams, and a Kubernetes-native shipper.