MCP Servers in Production: Patterns for Connecting AI Agents to Kubernetes and Cloud APIs¶

The first wave of AI agents talked to infrastructure through bespoke wrappers: a Python script that called kubectl, a Lambda that polled CloudWatch, a chat plugin that hard-coded an OpenAPI spec. That worked for demos. It collapsed the moment an agent needed to talk to two systems at once, or get rotated credentials, or appear in a SOC 2 audit.

Model Context Protocol (MCP) is the answer that won, and by 2026 it has the ecosystem to prove it. The Python SDK alone shipped v2.2.0 on 2026-09-07.¹ The spec was last revised 2025-06-18 and is now stable enough that AWS Labs, Microsoft, Google, Cloudflare, and Alibaba all ship vendor MCP servers against it.² OWASP published an MCP-specific Top 10 in 2025 because the attack surface is real enough to warrant one.³

This post is the field guide I wish someone had handed me twelve months ago. It covers the transport choices, the Kubernetes deployment patterns, the cloud-vendor landscape, authentication, security, multi-server gateways, observability, and the operational checklist that turns a working prototype into a production control plane.

Why MCP, and why now¶

MCP is a JSON-RPC protocol that exposes tools, resources, and prompts to a model. A server publishes a list of capabilities; a client (the agent runtime, an IDE, or a chatbot) discovers them, negotiates permissions, and invokes them. The model never sees raw HTTP, never holds a long-lived credential, and never learns the shape of an API — it just sees a typed function signature.

For platform engineers, three properties make MCP the right substrate:

1. Capability as a unit of governance. A tool is a function with a name, a JSON Schema for its inputs, and an authorization scope. You can review a tool the way you review a Helm chart.
2. Transport-agnostic. Stdio for local and sidecar; streamable HTTP for service-to-service; SSE where you need push. The application protocol is the same.
3. Auth is the framework's problem, not yours. The 2025-06-18 revision standardized OAuth 2.1 with PKCE for HTTP transports, so servers can delegate to your existing IdP instead of inventing a session model.²

The alternative — agents calling kubectl or boto3 directly — looks simpler for a single agent and falls apart at fleet scale, because every agent reimplements credential rotation, retry, audit, and rate limiting.

The transport decision¶

MCP defines three transports, and picking the wrong one is the most common mistake I see in production deployments.

The default for any production Kubernetes deployment should be streamable HTTP behind a service mesh. Stdio is for development and for the sidecar pattern where the agent pod spawns its own MCP server process.

The Cloudflare MCP server umbrella is a good reference for the multi-transport pattern: it ships per-product MCP servers (workers-mcp, containers-mcp, logpush-mcp, etc.) that all speak streamable HTTP, so a single agent can fan out to many product surfaces without juggling transports.⁴

Kubernetes deployment patterns¶

There are three patterns that work, and a fourth that looks like it works and doesn't.

Pattern 1: sidecar (development and per-pod isolation)¶

The MCP server runs as a sidecar container in the same pod as the agent. Communication is over localhost; the server inherits the pod's service account; no network policy is required for the data path.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: agent-with-k8s-mcp
spec:
  template:
    spec:
      serviceAccountName: agent-mcp
      containers:
        - name: agent
          image: my-agent:1.4.2
        - name: k8s-mcp
          image: ghcr.io/alexei-led/k8s-mcp-server:v1.4.2
          ports:
            - containerPort: 8080

This is the right pattern when one agent owns its own credentials and the blast radius of a compromised tool is the pod itself. The community alexei-led/k8s-mcp-server (v1.4.2, 2026-02-27) is the most-used reference, and reza-gholizade/k8s-mcp-server is the active fork with the most recent commits (last push 2026-09-19).⁵⁶

Pattern 2: in-cluster service (the production default)¶

The MCP server is a normal Deployment exposed through a ClusterIP service. Multiple agent deployments share it. Network policies restrict which namespaces can reach it. Authentication is mutual — typically a bearer token or mTLS, plus Kubernetes RBAC at the service-account level for any kubectl-style operations.

This is the pattern you want for shared infrastructure tools (log query, deployment promotion, secret rotation). The MCP server becomes a thin, audited facade over your existing control plane.

Pattern 3: sidecar over host network (avoid)¶

I have seen teams wire an MCP server to the host network and give it cluster-admin. Don't. The agent can read every secret in the cluster, every ConfigMap, every pod's env, with no per-call authorization. This pattern fails the OWASP MCP02 (Privilege Escalation via Scope Creep) and MCP09 (Shadow MCP Servers) checks out of the box.³

Pattern 4: gateway (multi-server fan-out)¶

The agent talks to a single MCP endpoint that fans out to multiple backend servers, each owning a different capability surface. The gateway handles auth, rate limiting, audit, and capability aggregation. This is the pattern Docker MCP Gateway, IBM Context Forge, and Archestra all implement.

I covered Docker MCP Gateway (v0.43.3, current as of 2026-09-16) and the broader landscape in Production AI Agents Need an MCP Gateway, so I won't re-litigate the comparison here. The short version: gateways pay off when you cross three or more capability surfaces, or when you need a single audit trail.⁷⁸⁹

The cloud-vendor MCP landscape¶

By 2026 every hyperscaler ships an MCP server. The interesting question is no longer "does it exist" but "which surface does it cover and is it still maintained."

The Azure lesson is worth dwelling on: check the archive status of every vendor MCP server before you build on top of it. The Microsoft pivot from azure-mcp to azure-skills took eight months and would have stranded any agent hard-wired to the old tool names.

Authentication: the part everyone underestimates¶

The 2025-06-18 MCP revision standardized OAuth 2.1 with PKCE for HTTP transports.² That gives you three production-ready patterns:

1. Bearer token from the agent's service account. The agent's pod carries a projected ServiceAccount token; the MCP server validates it against the Kubernetes TokenReview API; the server uses its own ServiceAccount to call Kubernetes. This is the cleanest pattern for in-cluster servers.
2. OAuth 2.1 with PKCE for external servers. The MCP server is an OAuth resource server; the agent (or the user behind it) is the resource owner; the IdP (Okta, Entra, Auth0) is the authorization server. This is the right pattern when the MCP server lives outside the cluster or when you need per-user attribution.
3. mTLS for high-trust internal links. Common when the MCP server is a sidecar or in the same mesh as the agent. SPIFFE IDs become the principal.

What you should not do: - Long-lived API keys embedded in MCP server config (OWASP MCP01) - Reusing the same bearer token across agents (OWASP MCP01 + MCP07) - Letting the model see the token in any context window or log line (OWASP MCP01)

A useful diagnostic: if your MCP server can restart without rotating any credentials, your auth model is broken.

Security: mapping OWASP MCP Top 10 to the K8s/cloud reality¶

The OWASP MCP Top 10 was published in 2025 and is the right baseline for any production rollout.³ Here's how each item shows up in a Kubernetes/cloud deployment and what to do about it.

• MCP01 (Token Mismanagement). Use projected ServiceAccount tokens with a short TTL (15 minutes is the default in Kubernetes 1.30+). Never log Authorization headers. Redact tokens in OpenTelemetry spans.
• MCP02 (Privilege Escalation via Scope Creep). Run the MCP server with a tightly scoped ServiceAccount. Use Kubernetes RBAC + a tool-level allow-list in the server. Audit the effective tools every quarter.
• MCP03 (Tool Poisoning). Pin tool descriptions to a known-good hash. Reject unsigned updates. Treat any third-party tool server the same way you'd treat an unsigned container image.
• MCP04 (Supply Chain). All MCP server images should come from a registry you control, signed (cosign or Sigstore), with SBOMs. The awslabs/mcp and Cloudflare releases are good models to copy.
• MCP05 (Command Injection). Treat every tool argument as untrusted input. Validate against the JSON Schema. Never let an agent string-interpolate into a shell command — use the typed SDK call instead.
• MCP06 (Prompt Injection via Context). Wrap retrieved content in <data> blocks and instruct the model to treat them as untrusted. This is the MCP equivalent of a Content Security Policy.
• MCP07 (AuthN/AuthZ). See the previous section. The single highest-impact change you can make.
• MCP08 (Lack of Audit/Telemetry). See the next section. Without this, you can't investigate an incident.
• MCP09 (Shadow MCP Servers). Run a NetworkPolicy that denies egress from agent namespaces to anything except your known MCP gateway. Alert on new ServiceAccounts that create Deployments.
• MCP10 (Context Over-Sharing). Don't pass entire ConfigMaps or large secret blobs into context. Filter at the tool layer; return only what the model needs.

Observability: treat MCP calls like any other RPC¶

An MCP call is just a JSON-RPC over a transport. Every observability backend you already use (OpenTelemetry, Prometheus, Datadog, Grafana) handles it out of the box. The pattern I recommend:

1. Span every tool invocation with the tool name, the duration, and the outcome. The MCP spec exposes hooks for exactly this.
2. Emit a metric for tool calls per second, per agent, per tool. Watch for cardinality explosion — the combination of agent × tool × resource × outcome can blow up a TSDB. The earlier post on AI Gateway cost control covers the cardinality control pattern in detail.¹⁵
3. Structured logs with the agent identity, the tool name, the tool arguments (redacted), and the result status. Never the result payload if it contains retrieved content.
4. Trace context propagated through the JSON-RPC meta field. The model context should be one span; each tool call should be a child.

I covered the broader pattern of instrumenting AI workloads with OpenTelemetry in LLM Observability with OpenTelemetry, which applies almost verbatim to MCP servers — the tool call is the LLM-adjacent RPC.

Multi-server fan-out: when one agent needs many surfaces¶

A real production agent rarely talks to one MCP server. It might need Kubernetes for deployment, AWS for cost data, GitHub for PRs, and PagerDuty for incidents. The naive approach — agent connects to all four directly — produces four auth contexts, four audit trails, and four rate-limit envelopes.

The gateway pattern consolidates this:

[agent] → [MCP Gateway] → ┬→ [k8s-mcp]
                          ├→ [aws-mcp]
                          ├→ [github-mcp]
                          └→ [pagerduty-mcp]

Docker MCP Gateway (v0.43.3) does this with a registry model: tools are declared in mcp.json, the gateway pulls them on demand, and the agent sees one endpoint.⁷ IBM Context Forge (v1.0.10, 2026-09-07) goes further and adds policy enforcement at the gateway layer.⁸ Archestra (platform-v1.3.65, 2026-09-19) is the newest entrant and has the strongest story around per-tool authorization.⁹

The right trigger to introduce a gateway: when you have three or more MCP servers in production or when you need a single audit trail for SOC 2 / ISO 27001. Below that, the operational overhead exceeds the benefit.

Operational checklist¶

The list I run against every MCP server before it gets a production label:

• Transport. Streamable HTTP behind a service mesh, unless there's a specific reason for stdio.
• Auth. OAuth 2.1 + PKCE for external; projected SA token + TokenReview for in-cluster. No long-lived secrets in config.
• Image. Pulled from a registry you control, signed, with SBOM. Pinned by digest, not tag.
• ServiceAccount. Tightly scoped. No cluster-admin. RBAC reviewed quarterly.
• NetworkPolicy. Egress from agent namespace restricted to known MCP endpoints. Egress from MCP server restricted to its backend APIs.
• Resource limits. Set explicitly. An MCP server that runs away is a denial-of-service vector against its backend.
• Health checks. Liveness probe on /healthz, readiness on /readyz. Tool-level readiness is harder; start with process-level.
• Observability. Spans on every tool call. Metrics with bounded cardinality. Structured logs with redaction.
• Rate limiting. Per-agent, per-tool. The earlier AI Gateway post covers this in depth.¹⁵
• Audit. Every tool call lands in an append-only log with agent identity, tool name, arguments (redacted), and outcome. Retention matches your compliance regime.
• Versioning. Pin the spec revision (2025-06-18 today). Track vendor release cadence — Azure's archive is a cautionary tale.
• Rollback. The previous version's container image is one kubectl rollout undo away. Test this in staging.
• Runbook. "MCP server X is returning 5xx" → which dashboard, which log query, which on-call. Write it before you need it.

Closing thought¶

MCP is the first protocol that lets platform engineers treat AI agents as first-class tenants of the infrastructure. The shape is familiar — auth, transport, observability, audit, supply chain — but the unit of governance is a tool, not a pod, and that's the shift that matters.

The patterns above are not theoretical. They're what's working in production at the teams I've worked with and watched ship over the last twelve months. The biggest predictor of success is treating MCP as infrastructure, not as a demo integration. The biggest predictor of failure is skipping the auth and audit story because the agent is "internal."

The vendor landscape will keep shifting — Azure's archive is the proof. Build on the protocol, not on any single server, and your agent fleet survives the next pivot.

Questions or discussion? Connect on LinkedIn, X, or drop me a line at pongsak@khamdee.com.

Discussion

Have thoughts on this post? Share them below — questions, corrections, or your own experience are all welcome.