Kong API Gateway on Kubernetes: a GitOps, Hybrid-Mode Reference¶

Run Kong so that a database outage can't take your APIs down, every config change goes through a merge request, and one gateway serves both public and internal traffic without mixing them. That's hybrid mode plus GitOps on Kubernetes. This post builds it on NKP (Nutanix Kubernetes Platform), with a companion project you can run on a laptop first.

Everything here is runnable. The Helm values, Gateway API manifests, Terraform/Atlantis config, secrets (HashiCorp Vault), monitoring (Prometheus + Grafana), and setup/test/day-2 scripts live in the kong-api-gateway project, all on community/open-source versions. Each section links to the file that implements it.

Verified against Kong Gateway 3.x and Kong Ingress Controller 3.5.x (June 2026).

The architecture¶

Config flows as code from GitLab; traffic flows through two data planes split by exposure:

  GitLab ──merge request──▶ Atlantis ──Terraform──▶ Admin API ──┐      S3 (TF state)
  (config as code)                                              │           ▲
                                                                ▼           │
  Konga (read-only GUI) ───GET───▶   Kong CONTROL PLANE ────────┴───────────┘
                                     (Admin API + KIC + Postgres, HA)
                                            │ pushes config over mTLS
                       ┌────────────────────┴────────────────────┐
                       ▼                                          ▼
            Kong DATA PLANE (external)                  Kong DATA PLANE (internal)
            north-south, public LoadBalancer            east-west, ClusterIP / private
                       │                                          │
            public client ──▶ ┘                 service A ──▶ ┘ (service-to-service)
                       ▼                                          ▼
              public microservices                       internal microservices

Three ideas carry the design:

• Hybrid mode splits Kong into a control plane (Admin API, talks to Postgres) and data planes (proxy traffic). The data planes cache their config on disk, so they keep serving even if the control plane or its database is down.¹
• External and internal data planes are separate deployments off the same control plane. Public traffic hits a LoadBalancer; internal traffic stays on a ClusterIP. Same config source, different blast radius.
• GitOps means no one runs curl against the Admin API by hand. Changes land through GitLab and Atlantis applies them with Terraform.⁴⁵

Why hybrid mode is the HA story¶

In a database-backed single tier, a Postgres outage means no config and often no proxy. In hybrid mode the proxy path is decoupled from the database entirely:

The HA failover test proves it: it scales the control plane to zero and shows the data planes still return 200.

Deploy on Kubernetes¶

One Helm chart (kong/kong) installed three times: control plane, external data plane, internal data plane.² Script: setup/10-helm-install.sh.

The control plane runs the Admin API, the Ingress Controller, and connects to Postgres (helm-values-control-plane.yaml):

env:
  role: control_plane
  database: postgres
  pg_host: kong-pg-rw.kong.svc.cluster.local
cluster:        { enabled: true, tls: { enabled: true } }    # CP<->DP mTLS
admin:          { enabled: true, type: ClusterIP }
proxy:          { enabled: false }                            # CP never proxies
ingressController: { enabled: true }

Each data plane runs DB-less, pulling config from the control plane (helm-values-dp-external.yaml):

env:
  role: data_plane
  database: "off"
  cluster_control_plane: kong-cp-kong-cluster.kong.svc.cluster.local:8005
proxy: { enabled: true, type: LoadBalancer }   # internal DP uses ClusterIP
admin: { enabled: false }
replicaCount: 3                                 # HA: survive a node loss

Postgres for the control plane should be HA too. The project uses the CloudNativePG operator for a 3-instance cluster with automatic failover (01-postgres-cnpg.yaml). Only the control plane touches it.

Routing with the Gateway API¶

Kong Ingress Controller is 100% conformant with the Kubernetes Gateway API, so routing is expressed as GatewayClass / Gateway / HTTPRoute instead of the legacy Ingress.³ Manifest: gateway-api.yaml.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: { name: httpbin, namespace: kong-apps }
spec:
  parentRefs: [{ name: kong-external, namespace: kong }]
  rules:
    - matches: [{ path: { type: PathPrefix, value: /httpbin } }]
      backendRefs: [{ name: httpbin, port: 80 }]

Attach a route to kong-external for public APIs, kong-internal for private ones.

Routing is not lifecycle: what Kong adds over the Gateway API¶

Here's the trap. The Gateway API does one job well: match a request and send it to a backend. HTTPRoute handles path and header matching, weighted traffic splitting, and a few filters (rewrite, redirect, header mutation). That is routing, and the spec deliberately stops there.

Managing an API is the other 90%: who can call it, how often, in what shape, and how you watch it. The Gateway API has no concept of a consumer, an API key, a quota, a cache, or a trace. If routing were enough, you wouldn't need a gateway, just a Service.

Kong fills that gap without replacing the Gateway API. You attach a KongPlugin to the same HTTPRoute with an annotation, so the route keeps its standard definition and gains gateway behavior on top:

apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata: { name: oidc-auth, namespace: kong-apps }
plugin: openid-connect
config:
  issuer: https://keycloak.example.com/realms/apis
  client_id: [kong]
  scopes: [openid, email]
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: orders
  namespace: kong-apps
  annotations:
    konghq.com/plugins: oidc-auth,rate-limit-orders   # Kong extends the route
spec:
  parentRefs: [{ name: kong-external, namespace: kong }]
  rules:
    - matches: [{ path: { type: PathPrefix, value: /orders } }]
      backendRefs: [{ name: orders, port: 80 }]

Manifest: k8s/kong-plugins.yaml. What each lifecycle stage needs, and who provides it:

Seven use cases that need the gateway, not just a route:

1. Authenticate once at the edge. Offload login to Keycloak or Okta with the openid-connect plugin, so no microservice handles tokens itself.
2. Per-consumer quotas. Free tier 100 requests/day, paid 100k, enforced by consumer group. The Gateway API has no notion of a caller to meter.
3. Protect a legacy backend. Rewrite modern JSON into the XML an old service expects with the transformer plugins, no backend change.
4. Canary a new version safely. The Gateway API splits 5% of traffic to v2; Kong adds the per-version metrics, headers, and auth so you can actually judge the rollout.
5. Reject bad input at the edge. Validate payloads against the OpenAPI spec with request-validator before they ever reach a pod.
6. Cache hot reads. proxy-cache serves repeat GETs from the gateway and sheds load off the upstream.
7. One observability surface. Prometheus metrics, OTel traces, and access logs for every API regardless of the language it's written in.

The Gateway API gives you the road. Kong adds the traffic lights, tolls, speed limits, and cameras. You need both, and on Kong the road is the Gateway API.

North-south and east-west: two postures, one platform¶

Not all API traffic is equal. North-south crosses your perimeter: external clients calling public APIs. East-west stays inside the cluster: your own services calling each other's internal APIs. They have opposite threat models, so they get separate data planes off the same control plane, which is exactly what the external and internal data planes are for.

  NORTH-SOUTH (external, public)              EAST-WEST (internal, service-to-service)

  Internet client                            Service A (orders)
       │ TLS                                       │  http://kong-internal/inventory
       ▼                                           ▼
  Kong EXTERNAL data plane (LoadBalancer)     Kong INTERNAL data plane (ClusterIP)
   • OIDC / key-auth                            • ACL (service identity)
   • strict per-consumer rate limit            • high rate limit (protect the callee)
   • request validation, CORS, IP allow        • retries, correlation id
       │                                           │
       ▼                                           ▼
  orders service (public API)                 inventory service (internal API)

Why split the directions:

• Blast radius and SLA. A public surge or DDoS lands on the external data plane; internal service-to-service calls keep flowing on the internal one. You can give them different autoscalers and even different node pools.
• Security posture. North-south is zero-trust: authenticate every request, rate-limit per consumer, validate the body, check CORS. East-west trusts service identity (ACL plus NetworkPolicy, or mesh mTLS) and tunes for throughput and retries instead of public auth.
• Exposure. The external data plane is a LoadBalancer with a public IP; the internal one is ClusterIP, reachable only inside the cluster, with a NetworkPolicy so upstreams accept traffic only through the gateway.

Wiring example¶

Full manifest: k8s/traffic-patterns.yaml. A public orders API on the external gateway:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: orders
  namespace: kong-apps
  annotations:
    konghq.com/plugins: oidc-auth,rate-limit-strict,validate-orders,cors
spec:
  parentRefs: [{ name: kong-external, namespace: kong }]   # north-south
  hostnames: ["api.example.com"]
  rules:
    - matches: [{ path: { type: PathPrefix, value: /orders } }]
      backendRefs: [{ name: orders, port: 80 }]

An internal inventory API on the internal gateway, called by orders over east-west:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: inventory
  namespace: kong-apps
  annotations:
    konghq.com/plugins: acl-internal,rate-limit-internal
spec:
  parentRefs: [{ name: kong-internal, namespace: kong }]   # east-west, no public IP
  rules:
    - matches: [{ path: { type: PathPrefix, value: /inventory } }]
      backendRefs: [{ name: inventory, port: 80 }]

Callers reach it at http://kong-dp-internal-kong-proxy.kong.svc/inventory, never a public address. A NetworkPolicy makes the inventory pods accept traffic only from the kong namespace, so every internal call goes through the gateway:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: inventory-gateway-only, namespace: kong-apps }
spec:
  podSelector: { matchLabels: { app: inventory } }
  policyTypes: [Ingress]
  ingress:
    - from: [{ namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: kong } } }]

For transparent per-hop mTLS on all east-west traffic, layer a service mesh (Kong Mesh / Kuma is open source) under the internal gateway: the gateway handles API-level policy, the mesh handles connection-level identity.

GitOps: config as code¶

No hand edits. Config lives in GitLab; Atlantis plans on every merge request and applies after approval, using the Terraform Kong provider against the Admin API, with state in S3. Files: gitops/terraform/ and gitops/atlantis.yaml.

resource "kong_service" "httpbin" {
  name = "httpbin"
  url  = "http://httpbin.kong-apps.svc.cluster.local:80"
}
resource "kong_route" "httpbin" {
  paths      = ["/httpbin"]
  service_id = kong_service.httpbin.id
}
resource "kong_plugin" "rate_limiting" {
  name        = "rate-limiting"
  service_id  = kong_service.httpbin.id
  config_json = jsonencode({ minute = 60, policy = "local" })
}

Two notes on tooling:

• decK is the declarative alternative and the right tool for backups and drift detection: deck gateway dump, deck gateway diff, deck gateway sync.⁶ Same state in Kong's native YAML (gitops/deck/kong.yaml).
• Konga is the read-only GUI in this design, for visualizing config. It's a community tool, so management goes through Terraform/decK, not the GUI. For an official UI, use Kong Manager (Enterprise) or Konnect.

Plugins: auth, rate limiting, metrics¶

Plugins are where a gateway earns its keep. The setup script enables key auth, rate limiting, a correlation id, and Prometheus metrics (setup/40-plugins.sh). The plugin test proves they enforce:

1) no API key        -> 401   (key-auth rejects anonymous)
2) valid key         -> 200
3) past 5 req/min    -> 429    (rate-limiting kicks in)

If those three assertions pass, your auth and rate limiting are real, not just configured.

Secrets without plaintext: HashiCorp Vault¶

Hardcoding a database password or an OAuth client secret in Kong config (and therefore in Git) is the most common way a gateway leaks credentials. Kong solves it with secret references: store the value in HashiCorp Vault, register Vault as a backend, and reference it as {vault://...} in any config field. Kong fetches the secret at request time, so the expanded value never lands in config, Git, or etcd.⁷

On Kubernetes, Kong authenticates to Vault with its own ServiceAccount through Vault's Kubernetes auth method, so there's no static token to leak (k8s/kong-vault.yaml):

apiVersion: configuration.konghq.com/v1alpha1
kind: KongVault
metadata: { name: my-hcv }
spec:
  backend: hcv
  prefix: my-hcv
  config:
    host: vault.vault.svc.cluster.local
    mount: secret
    kv: v2
    auth_method: kubernetes      # uses the Kong pod's ServiceAccount JWT, no static token
    kubernetes_role: kong

Then reference the secret anywhere a field is referenceable, the Postgres password, an OIDC client secret, a TLS key, or a plugin credential:

{vault://my-hcv/demo/apikey}

Vault itself runs as community OSS in HA mode with integrated Raft storage (k8s/vault-helm-values.yaml). The vault test proves resolution: it injects {vault://my-hcv/demo/apikey} as a request header and confirms the upstream receives the resolved value, not the literal reference.

Monitoring with Prometheus and Grafana¶

A gateway you can't see is a gateway you can't operate. The Prometheus plugin (enabled cluster-wide in setup/40-plugins.sh) exposes request rate, status codes, Kong and upstream latency, bandwidth, and upstream health on a /metrics endpoint.⁸

On Kubernetes, a ServiceMonitor feeds kube-prometheus-stack (Prometheus + Grafana OSS), and the official dashboards load by id (k8s/monitoring/, setup/60-monitoring.sh):

spec:
  selector: { matchLabels: { app.kubernetes.io/name: kong } }
  endpoints:
    - targetPort: 8100      # Kong status listener
      path: /metrics
      interval: 15s

Import Grafana dashboards 7424 (Kong) and 15662 (Kong Ingress Controller). Watch four signals: request rate, 5xx rate, p99 latency split into Kong vs upstream (so you know whether the gateway or the service is slow), and rate-limit rejections. The metrics test confirms the series exist after a little traffic.

Locally, docker compose up already runs Prometheus and Grafana with a slim Kong dashboard provisioned, so you can see metrics at localhost:3000 before touching a cluster.

Ship API logs to the Elastic Stack¶

Metrics tell you how much; logs tell you what happened on a single request. Kong is the front door, so its logs are your API access record. Send them to the centralized Elastic Stack from the previous post, Centralize Log Solution with the Elastic Stack. Use both paths:

1. Access logs (passive). Kong's proxy access logs go to stdout, so the Filebeat DaemonSet from that post already collects them once you point it at the kong namespace. Kong is your ingress, so the "Kubernetes ingress logs" use case in that post literally is Kong's traffic.

2. Structured API logs (active, recommended). The http-log plugin posts a JSON object per request with the fields raw access logs lack: the matched service and route, the consumer, and Kong-vs-upstream latency. Point it at the Logstash HTTP input from the logging post (k8s/kong-logging.yaml):

apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata: { name: http-log, labels: { global: "true" } }
plugin: http-log
config:
  http_endpoint: http://logstash.logging.svc.cluster.local:8080
  method: POST
  content_type: application/json
  queue: { max_batch_size: 10 }

Kong batches and POSTs newline-delimited JSON. Logstash parses it into a logs-kong.proxy-default data stream, which LogsDB and ILM manage automatically (anything matching logs-*). The receiving pipeline is logstash/kong-http-log.conf; drop it into the Logstash tier from the logging post.

That completes the loop: per-API logs in Kibana, request-rate and latency in Grafana, secrets in Vault, all from one gateway.

Day-2 operations¶

Run it locally first¶

git clone https://github.com/pkhamdee/kong-api-gateway
cd kong-api-gateway/compose
cp .env.example .env            # set passwords
docker compose up -d            # Kong + Postgres + Konga + httpbin
cd ..
export ADMIN_URL="http://localhost:8001" PROXY_URL="http://localhost:8000"
./setup/run-all.sh              # service, route, plugins
./test/smoke-test.sh && ./test/route-test.sh && ./test/plugin-test.sh

The local stack is a single DB-backed node so it fits on a laptop. The Kubernetes manifests run the full hybrid topology. The Kong config and plugins are identical against both.

Summary¶

The Kong platform that scales and stays up:

• Hybrid mode: control plane with HA Postgres, data planes that cache config and survive a CP/DB outage.
• Two data planes by traffic direction: external for north-south (public, zero-trust), internal for east-west (service-to-service, ACL + NetworkPolicy), one config source.
• Gateway API for routing, not legacy Ingress.
• GitOps: GitLab + Atlantis + Terraform to the Admin API, state in S3. No hand edits.
• decK for backups and drift; Konga read-only for visibility.
• Plugins enforced: key auth, rate limiting, Prometheus. Prove it with the plugin test.
• Secrets in Vault: referenced as {vault://...}, fetched at runtime. No plaintext credentials in config or Git.
• Monitoring wired: Prometheus scrapes /metrics, Grafana dashboards 7424 / 15662. Watch request rate, 5xx, p99, rate-limit rejections.
• API logs to Elastic: http-log plugin to Logstash, into a logs-kong.* data stream (see the logging post).
• HA proven: scale the control plane to zero and confirm traffic still flows.

Decouple the proxy from the database, split traffic by exposure, and drive every change through Git. The gateway becomes boring, which is exactly what you want from a gateway.

Questions or discussion? Connect on LinkedIn, X or reach out via email.

Discussion

Have thoughts on this post? Share them below — questions, corrections, or your own experience are all welcome.