Skip to content

Self-Host an AI Gateway on Kubernetes: Routing, Caching, and Cost Control with Envoy

One LLM provider is a weekend project. Five providers, two self-hosted inference clusters, three regions, and a per-team budget is a platform. The path between them is an AI gateway — and Envoy is the only data plane with a first-class AI filter shipped upstream. Run it on Kubernetes the way you already run any other gateway, and you get a single control plane for every model call in the company.

What an AI gateway actually does

An AI gateway is a thin policy layer in front of LLM endpoints. It looks like Envoy, talks like Envoy, but understands the LLM request shape. Five jobs matter in production:

Job What it solves
Routing Send gpt-4o to OpenAI, claude-sonnet-4.5 to Anthropic, qwen3-coder to your vLLM cluster, all behind one OpenAI-compatible endpoint.
Failover If OpenAI's API returns 5xx, fall over to Anthropic or to a self-hosted mirror of the same model.
Semantic caching Identical prompts return cached completions. Cuts cost and TTFT on high-traffic identical queries.
Token accounting Record input, output, and cached token counts per request, per user, per team. Push to your observability backend.
Cost ceilings Cap monthly spend per team, per model, per route. Reject (or rate-limit) requests that exceed the budget.

Everything else — retries, TLS, observability, Kubernetes-native deployment — you already get from Envoy.

The data plane: Envoy with the AI filter

Envoy 1.39.1 (released 2026-08-27) ships the AI extension filter upstream.1 The filter understands the OpenAI, Azure OpenAI, AWS Bedrock, Google Gemini, and Vertex AI request and response shapes. It runs as an HttpFilter on a Route and rewrites the upstream cluster without touching the application.

What the AI filter does on each request:

  • Parse the JSON body to extract the model name, the message array, and the streaming flag.
  • Inject provider-specific headers (Authorization: Bearer …, api-key: …, x-api-key: …).
  • Capture usage stats (prompt_tokens, completion_tokens, total_tokens) and write them to Envoy's dynamic metadata.
  • Stream the response back to the client chunk by chunk, accounting for usage on completion.

The application code does not change. The upstream endpoints become opaque to the developer.

The control plane: Agent Router (formerly Envoy AI Gateway)

The project used to be called Envoy AI Gateway. As of v1.1.0 (released 2026-08-21), it has been renamed Agent Router under the Agentic AI Foundation.2 It's the same code, same maintainers, same Apache-2.0 license — and the CRDs and CLI haven't been renamed:

AIGatewayRoute            aigateway.envoyproxy.io/v1alpha1
AIServiceBackend          aigateway.envoyproxy.io/v1alpha1
BackendSecurityPolicy     aigateway.envoyproxy.io/v1alpha1

The Helm chart, container images, and the aigw CLI are unchanged. The repo and docs moved:

  • envoyproxy/ai-gateway → theagentrouter/agent-router
  • aigateway.envoyproxy.io → theagentrouter.ai

If you deploy today, write the manifests with the aigateway.envoyproxy.io API group. They still work, and a future migration will preserve them.

Reference architecture

                  ┌──────────────────────┐
                  │  Apps / Agents        │
                  │  (OpenAI SDK, etc.)   │
                  └──────────┬───────────┘
                             │  POST /v1/chat/completions
                             ▼
                  ┌──────────────────────┐
                  │  Envoy (with         │
                  │  AI ext_proc filter) │
                  │  - parse request     │
                  │  - inject auth       │
                  │  - cache lookup      │
                  │  - emit usage        │
                  └──────┬───────────────┘
                         │ route by model + header
        ┌────────────────┼───────────────────────┐
        ▼                ▼                       ▼
  ┌──────────┐     ┌──────────┐             ┌──────────┐
  │ OpenAI   │     │ Anthropic│             │ vLLM on  │
  │ upstream │     │ upstream │             │ k8s (in  │
  │          │     │          │             │ cluster) │
  └──────────┘     └──────────┘             └──────────┘
        │                │                       │
        └────────────────┼───────────────────────┘
                         │  OTLP / metrics scrape
                         ▼
                  ┌──────────────────────┐
                  │  OTel Collector →    │
                  │  Prometheus / Tempo  │
                  └──────────────────────┘

Envoy is the data plane. Agent Router is the Kubernetes-native control plane that compiles CRDs into Envoy configuration. The OpenTelemetry pipeline is the same one your other services already use.

Deploying on Kubernetes

The v1.0.0 (2026-06-23) release was the first generally available version.3 Install via Helm:

helm repo add envoy-ai-gateway https://aigateway.envoyproxy.io/charts
helm install ai-gateway envoy-ai-gateway/ai-gateway-helm \
  --namespace envoy-ai-gateway-system \
  --create-namespace

This deploys the CRDs, the controller, and a default Envoy Gateway (InferenceGateway).

A real AIGatewayRoute

This is the kind of manifest you put in version control. It routes qwen3-coder-30b-a3b to your self-hosted vLLM cluster and gpt-4o-mini to OpenAI, applies a per-team token quota, and emits usage metadata for the OTel pipeline.

apiVersion: aigateway.envoyproxy.io/v1alpha1
kind: AIGatewayRoute
metadata:
  name: chat-routes
spec:
  parentRefs:
    - name: ai-gateway
      kind: Gateway
  hostnames: ["llm.example.com"]
  rules:
    - matches:
        - headers:
            - type: Exact
              name: x-ai-eg-model
              value: qwen3-coder-30b-a3b
      backendRefs:
        - name: self-hosted-vllm
    - matches:
        - headers:
            - type: RegularExpression
              name: x-ai-eg-model
              value: "^gpt-4o.*"
      backendRefs:
        - name: openai
  llmRequestCosts:
    - metadataKey: llm_input_token
      type: InputToken
    - metadataKey: llm_output_token
      type: OutputToken
    - metadataKey: llm_total_token
      type: TotalToken

The backend

apiVersion: aigateway.envoyproxy.io/v1alpha1
kind: AIServiceBackend
metadata:
  name: self-hosted-vllm
spec:
  schema:
    name: OpenAI
  backendRef:
    name: vllm-cluster
    kind: Backend
    port: 8000
---
apiVersion: aigateway.envoyproxy.io/v1alpha1
kind: AIServiceBackend
metadata:
  name: openai
spec:
  schema:
    name: OpenAI
  backendRef:
    name: openai-upstream
    kind: Backend
    port: 443
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: Backend
metadata:
  name: openai-upstream
spec:
  endpoints:
    - fqdn:
        hostname: api.openai.com
        port: 443

Security policy for the API key

The BackendSecurityPolicy references a Kubernetes Secret. The controller mounts the secret and the AI filter sends Authorization: Bearer … on every request. No app code touches the key.

apiVersion: aigateway.envoyproxy.io/v1alpha1
kind: BackendSecurityPolicy
metadata:
  name: openai-key
spec:
  targetRefs:
    - group: aigateway.envoyproxy.io
      kind: AIServiceBackend
      name: openai
  type: APIKey
  apiKey:
    secretRef:
      name: openai-credentials

The openai-credentials Secret has OPENAI_API_KEY in its data map.

The three things people get wrong

1. Forgetting to extract the model header for routing

The AI filter rewrites the request, but routing decisions happen before the body is parsed. You need to populate the x-ai-eg-model header from the JSON body so Envoy can match on it. The pattern in the manifest above (type: Exact / RegularExpression matches on x-ai-eg-model) is the right shape; the trap is forgetting to configure the filter to populate that header.

The recommended pattern is to let the AI filter do the work in ext_proc:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: ai-ext-proc
spec:
  provider:
    type: Kubernetes
  bootstrap:
    type: Merge
    merge:
      resources:
        - kind: Cluster
          ...
  extProc:
    - name: ai-gateway
      backendRefs:
        - name: ext-proc-service
          port: 1063

This pushes body parsing into Envoy and the routing decision can use model name as a header.

2. Caching the wrong thing

Semantic caching only works for genuinely identical prompts at the LLM level. The naive approach of caching by (prompt, model) is brittle:

  • Streamed responses are hard to cache as a complete unit.
  • Tool calls are stateful — caching them across requests is unsafe.
  • Producing different outputs at temperature > 0 is the default behaviour.

The pattern that holds up:

  • Cache only on exact (model, prompt-text, temperature=0, no-tools).
  • Use the OpenAI seed header to make the request deterministic on the model side.
  • Set a short TTL (60–300 seconds). Beyond that, even identical prompts may have moved.

The Agent Router project supports a Redis-backed semantic cache that does prompt-hash matching internally. If you cache, wire it to Redis with eviction policy set, and never cache tool calls.

3. Cost accounting without cardinality control

The fastest way to crash your Prometheus is to emit gen_ai.usage.input_tokens per token per request per user per team. That's at least 5 dimensions and can blow out the time-series budget.

The fix: aggregate at the gateway. Use the cost-control layer to write one counter per request, not one per token. Token counts stay in spans, not metrics.

# OTel collector processor to roll up token costs
processors:
  attributes/cost:
    actions:
      - key: gen_ai.usage.input_tokens
        action: delete
      - key: gen_ai.usage.output_tokens
        action: delete
  transform/cost:
    metric_statements:
      - context: metric
        statements:
          - set(name, "llm.request.count") where name == "gen_ai.client.token.usage"

Roll up token totals per request in the collector, never per token.

The thing the docs don't emphasise: two-tier gateways

Agent Router's docs describe a two-tier gateway pattern that matters once you self-host inference.2

  • Tier 1 gateway. Centralised. Owns authentication, top-level routing, and global rate limits. This is what app teams see.
  • Tier 2 gateway. Per-cluster. Endpoint picker support, prefix-cache routing, model-specific fallback. Sits in front of the vLLM workers.

The Gateway API Inference Extension (covered in earlier posts) is the tier 2 story. Agent Router is the tier 1 story. Two tiers, two control planes, one OpenAI-compatible URL per model.

If you run tier 1 only, you have central auth and quotas but no specialised inference routing. If you run tier 2 only, you have smart model routing but no platform-level policy. Run both. They aren't competing options.

Operational checklist for self-hosted AI gateway on Kubernetes

  • Version pin. Lock envoy-ai-gateway/ai-gateway-helm to a specific chart version. The v1.0.0 GA (2026-06-23) is the safe starting point.
  • Resource limits. The Envoy proxy is memory-hungry when streaming. 1 CPU / 2 Gi per pod as a baseline; right-size from there.
  • HA. Three replicas minimum behind a Service with anti-affinity. The gateway is in the critical path.
  • TLS. Bring your own cert via cert-manager + Gateway API TLS. The AI filter doesn't do TLS itself.
  • OTel. Wire OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_SERVICE_NAME to your existing collector. Don't build a new observability stack.
  • Sealed secrets. API keys go in SealedSecret (Bitnami) or External Secrets Operator. Never in plain Secret in git.
  • Rate limiting. Use Redis-backed Envoy rate limit service. Per-team limits, per-model limits, and a global ceiling.
  • Cost dashboards. One Grafana dashboard per team. Token spend per model, per day. Records stop you from being surprised at the end of the month.
  • Audit log. Every request gets a structured log line: request ID, user, model, token counts, status, latency. Ship to your existing log pipeline.
  • Disaster test. Pull the OpenAI credential and verify failover to the self-hosted backend works. Document the runbook.

When NOT to run your own AI gateway

Three situations where the operational cost of running it yourself is wrong:

  1. One team, one model, < 10M tokens/month. Point your app directly at the provider. Add the gateway when the second team or second provider shows up.
  2. You already pay for managed Kong, Apigee, or Portkey. Those have AI-specific extensions. Use them until you actually need Envoy's AI filter.
  3. You're optimising for time-to-first-token on a single model. The gateway adds 1–5 ms per request. Acceptable for production; wasteful if you're shipping a one-team PoC.

Summary

  • Envoy 1.39.1 (released 2026-08-27) ships the AI extension filter upstream.
  • Agent Router (formerly Envoy AI Gateway) v1.1.0 (released 2026-08-21) is the Kubernetes-native control plane, Apache-2.0, with AIGatewayRoute, AIServiceBackend, and BackendSecurityPolicy CRDs.
  • Five jobs: routing, failover, semantic caching, token accounting, cost ceilings.
  • The CRD shape is real: route by x-ai-eg-model, attach per-team BackendSecurityPolicy, emit llmRequestCosts for OTel.
  • Don't cache streamed responses, tool calls, or non-zero temperature.
  • Roll token counts up at the collector, not at the metric.
  • Two-tier pattern: tier 1 central gateway, tier 2 inference gateway with endpoint picker.
  • HA from day one. The AI gateway is in the critical path; treat it like one.

You wouldn't run naked HTTP without a proxy. Don't run naked LLM endpoints without one either.

Questions or discussion? Connect on LinkedIn, X, or email.

  1. Envoy, Releases — v1.39.1 published 2026-08-27. ↩

  2. Agent Router, Repository README — formerly Envoy AI Gateway, Agentic AI Foundation project; Apache-2.0; CRDs and CLI names unchanged. ↩↩

  3. Agent Router, Releases — v1.0.0 GA published 2026-06-23; v1.0.0 GA on the day before this post. ↩

  4. Agent Router, README — two-tier gateway pattern — describes Tier 1 centralised gateway (auth, top-level routing, global rate limits) and Tier 2 inference gateway (endpoint picker, prefix-cache routing, model fallback). ↩

  5. Agent Router, API types — AIGatewayRoute, AIServiceBackend, BackendSecurityPolicy, MCPRoute, QuotaPolicy, GatewayConfig CRDs in the aigateway.envoyproxy.io/v1alpha1 API group. ↩

Discussion

Have thoughts on this post? Share them below — questions, corrections, or your own experience are all welcome.