Cilium and eBPF: the Networking Layer Under Your Cluster¶

Your cluster routes every packet through iptables, and at a few thousand Services that chain becomes a linear scan the kernel walks per connection. Cilium throws that out and runs networking, policy, and observability as eBPF programs in the kernel instead — no sidecars, no iptables, no agent in the data path. This post walks the whole stack: how a packet even finds another pod, installing Cilium as a kube-proxy replacement, watching real traffic with Hubble, locking the cluster down with identity-based policy, stretching it across clusters with Cluster Mesh, and dropping the service-mesh sidecars — all copy-paste.

Verified against Cilium 1.18.x and Kubernetes 1.33 (June 2026). Cilium is a CNCF graduated project and is what GKE Dataplane V2 runs under the hood, so this isn't bleeding-edge — it's what the big managed clusters already ship.¹

Why iptables is the problem¶

kube-proxy programs Service load balancing as iptables rules. Every Service times every endpoint becomes a rule, and the kernel evaluates them as a sequential chain. Add Services and the chain grows; connection setup gets slower across the board.

# On a node running kube-proxy, this is what scales linearly:
iptables-save | grep -c KUBE-SVC

The list-based design is the well-known one. It's also why network policy enforced at the IP/port level breaks the moment pods reschedule and get new IPs — the policy is pinned to an address that no longer means anything.

eBPF replaces the whole model. Cilium attaches programs directly to network interfaces in the kernel. Service lookup is a hash-map, not a chain scan. Policy is enforced against a stable identity derived from pod labels, not an IP that churns on every reschedule.²

Reported numbers from production write-ups land around 40% lower latency and ~60% less CPU at scale versus iptables. Treat those as illustrative — they depend heavily on Service count and traffic shape — but the direction is consistent everywhere.¹³

How pods get networked: the CNI story¶

Before the eBPF wins make sense, it helps to see the problem CNIs were invented to solve. Build it up from nothing.

Three VMs, each running containers on a local bridge. Give every bridge the same subnet — say 192.168.1.0/24 — and IPs collide the moment two hosts hand out 192.168.1.5. So give each host its own subnet: .1.0/24, .2.0/24, .3.0/24. Now the addresses are unique, but a new problem appears: how does a packet from etcd1 on host A reach etcd2 on host B? Host A's kernel has no idea host B's container subnet exists.

        Host A  eth0:10.0.30.11            Host B  eth0:10.0.30.12
   ┌──────────────────────────┐      ┌──────────────────────────┐
   │  bridge 192.168.2.0/24   │      │  bridge 192.168.3.0/24   │
   │    ┌───────┐             │  ??  │    ┌───────┐             │
   │    │ etcd1 │ .2.10  ─────┼──────┼──▶ │ etcd2 │ .3.20       │
   │    └───────┘             │      │    └───────┘             │
   └────────────┬─────────────┘      └────────────┬─────────────┘
                └──────────── 10.0.0.0/8 ─────────┘
        Host A has no route to 192.168.3.0/24  ✗  — who teaches it?

There are many ways to solve that — and that's the point. Enough ways that some people wrote a spec: CNI, the Container Networking Interface — the contract between the orchestrator (Kubernetes) and whatever wires the network. Kubernetes calls a CNI plugin to set up each pod's networking, and the plugin owns the how.⁶

Kubernetes hands the CNI four hard rules:

• Pod-to-pod without NAT — every pod gets a unique IP and can reach any other pod directly, same node or not.
• Node-to-pod — the kubelet and node daemons can reach pods on that node.
• Service-to-pod — Service virtual IPs resolve to backing pods.
• Ingress/egress — traffic gets in and out of the cluster.

Three separate networks are in play at once, and conflating them is where confusion starts:

A classic CNI like Flannel wires it like this: each pod gets a veth pair into a cni0 bridge, and cross-node traffic is VXLAN-encapsulated — wrapped in a normal node-to-node UDP packet on the source node, unwrapped on the destination. That's the overlay. The host network never has to understand pod addresses; it only routes node-to-node:

  etcd1 ─▶ veth ─▶ cni0 ─▶ VXLAN encap ─▶ eth0 ─┐
  10.200.1.4              (Host A)               │   host network
                                                 │   10.0.0.0/8
  etcd2 ◀─ veth ◀─ cni0 ◀─ VXLAN decap ◀─ eth0 ◀─┘   (sees node IPs only)
  10.200.2.6              (Host B)

  on the wire:  [ outer 10.0.30.11 → 10.0.30.12 | inner 10.200.1.4 → 10.200.2.6 | data ]
                └─ node-to-node, what the network routes ─┘ └─ pod-to-pod, hidden inside ─┘

Then kube-proxy programs iptables to DNAT a Service VIP onto a real pod IP — the linear chain from the section above.

Cilium collapses two of those hops. The Service lookup becomes an eBPF hash-map read instead of an iptables chain, and the encapsulation becomes optional. Which is the next decision.

Encapsulating vs native routing¶

Cilium runs the pod overlay in one of two modes, and the choice is dictated by your underlying network:⁷

Encapsulating mode is the safe default — it asks nothing of your network because every pod packet rides inside an ordinary UDP packet between node IPs. The cost is ~50 bytes of header per packet, marginal at 1500-byte frames but real, and worse without jumbo frames.

Native routing drops the wrapper for lower overhead and latency, but now the network has to know how to route pod addresses. On a flat L2 segment, Cilium can tell every node about each PodCIDR directly. Otherwise you run a BGP daemon so the fabric or an external router learns the routes. Faster, but it's no longer zero-config.

The difference is what hits the wire:

  Encapsulating:  [ outer IP/UDP/VXLAN  node→node ][ inner IP  pod→pod ][ data ]
                   └──── +50 bytes ────┘   network routes the OUTER (node) IPs

  Native routing: [ inner IP  pod→pod ][ data ]
                   network must route the POD IPs itself  (flat L2 or BGP)

Install Cilium as a kube-proxy replacement¶

The point is to run without kube-proxy at all. Bring up the cluster with kube-proxy disabled, then let Cilium own Service load balancing.

Prerequisites:

• Linux kernel 5.10+ on every node (RHEL 8.6 equivalent or newer).³
• Cluster created without kube-proxy (kubeadm init --skip-phases=addon/kube-proxy, or delete the kube-proxy DaemonSet on an existing cluster).
• helm 3.x and the Cilium CLI installed.

# Install the Cilium CLI (Linux x86_64)
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
curl -L --fail -o cilium.tar.gz \
  "https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz"
sudo tar xzvf cilium.tar.gz -C /usr/local/bin && rm cilium.tar.gz

Install Cilium with kube-proxy replacement on. You must give it the API server address, because with no kube-proxy there's no kubernetes Service backing yet:

helm repo add cilium https://helm.cilium.io/
helm repo update

helm install cilium cilium/cilium --version 1.18.1 \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=<API_SERVER_IP> \
  --set k8sServicePort=6443 \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true

kubeProxyReplacement=true is the whole game — Cilium now handles ClusterIP, NodePort, LoadBalancer, and HostPort in eBPF.²

Verify it actually replaced kube-proxy¶

Don't trust the install log — prove it.

cilium status --wait

# Confirm kube-proxy replacement is active and there's no kube-proxy DaemonSet
kubectl -n kube-system exec ds/cilium -- cilium-dbg status | grep KubeProxyReplacement
# KubeProxyReplacement:   True   [eth0 ...]

kubectl -n kube-system get ds kube-proxy
# Error from server (NotFound): daemonsets.apps "kube-proxy" not found

If KubeProxyReplacement: True and the DaemonSet is gone, every Service in the cluster is now load-balanced by eBPF. Run the built-in connectivity test before you go further:

cilium connectivity test

Hubble: see every flow without instrumenting anything¶

This is the part that sells Cilium to people who don't care about CNIs. Because the eBPF programs already see every packet, flow visibility is free — no sidecars, no app changes, no service mesh.⁴

cilium hubble enable --ui
cilium hubble port-forward &

Watch live traffic, filtered to what you care about:

# Every flow into the payment service, in real time
hubble observe --namespace prod --to-pod payment --follow

# Only the DROPPED flows — instantly answers "what is my NetworkPolicy blocking?"
hubble observe --verdict DROPPED --follow

# DNS lookups across the cluster
hubble observe --protocol dns --follow

hubble observe --verdict DROPPED is the single most useful command here. When a policy breaks an app, you normally guess; with Hubble you watch the exact source identity, destination, and port being denied. The Hubble UI (cilium hubble ui) draws the same data as a live service map.

If you're already shipping logs off-cluster, Hubble flows export as JSON and slot straight into the kind of Elastic pipeline I built in the centralized logging post.

Identity-based network policy you can actually maintain¶

Standard Kubernetes NetworkPolicy works on Cilium. But CiliumNetworkPolicy is where the identity model pays off — and where you get L7 rules iptables can't express.

Start with the rule every cluster should have and almost none do: default-deny. Until you apply this, every pod can talk to every other pod.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: default-deny
  namespace: prod
spec:
  endpointSelector: {}      # all pods in the namespace
  ingress:
    - {}                    # deny all ingress not explicitly allowed

Now allow exactly what the app needs. This says: pods labelled app=api may reach app=payment only on TCP 8080, and nothing else can:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-to-payment
  namespace: prod
spec:
  endpointSelector:
    matchLabels:
      app: payment
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: api
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP

The selector is app: api, not an IP range. Pods can die, reschedule, and scale to 50 replicas — the policy holds because it's bound to the identity, not the address.

L7: restrict by HTTP method and path¶

Here's the rule you cannot write with plain NetworkPolicy. Let api call payment, but only GET /charges and POST /charges — block everything else at the HTTP layer, in-kernel:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-to-payment-l7
  namespace: prod
spec:
  endpointSelector:
    matchLabels:
      app: payment
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: api
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:
              - method: "GET"
                path: "/charges"
              - method: "POST"
                path: "/charges"

A compromised api pod now can't hit /admin on the payment service even though it's allowed on port 8080. That's L7 segmentation without a service mesh.

Lock down DNS egress¶

Stop pods from resolving arbitrary domains — a cheap, high-value control against exfiltration:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: egress-dns-allowlist
  namespace: prod
spec:
  endpointSelector:
    matchLabels:
      app: api
  egress:
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
          rules:
            dns:
              - matchPattern: "*.svc.cluster.local"
              - matchName: "api.stripe.com"

The api pods can resolve in-cluster names and api.stripe.com. Any other lookup is dropped — and you'll see it in hubble observe --protocol dns --verdict DROPPED.

Egress Gateway: a stable source IP for the outside world¶

Policy controls what a pod may reach. Egress Gateway controls what IP the destination sees it coming from. By default a pod's traffic leaves through whatever node it happens to run on, SNAT'd to that node's IP — so the source address churns as pods reschedule across nodes.

That breaks the moment you talk to something with an IP allowlist: a legacy database, a partner API, an on-prem firewall. They want one predictable source IP, and "any of my 40 nodes, whichever the scheduler picked" isn't it.

Cilium routes selected egress traffic through designated gateway nodes and SNATs it to a fixed egress IP. Enable it at install (needs kube-proxy replacement and BPF masquerade):¹⁰

helm upgrade cilium cilium/cilium --version 1.18.1 \
  --namespace kube-system --reuse-values \
  --set egressGateway.enabled=true \
  --set bpf.masquerade=true

Then a CiliumEgressGatewayPolicy says "traffic from these pods to that network leaves via this node, as this IP":

apiVersion: cilium.io/v2
kind: CiliumEgressGatewayPolicy
metadata:
  name: api-egress-via-gateway
spec:
  selectors:
    - podSelector:
        matchLabels:
          app: api
          io.kubernetes.pod.namespace: prod
  destinationCIDRs:
    - "203.0.113.0/24"        # the partner's network
  egressGateway:
    nodeSelector:
      matchLabels:
        egress-node: "true"   # label your gateway node(s)
    egressIP: "192.0.2.10"    # the single IP the partner allowlists

Now prod/api pods reaching 203.0.113.0/24 always egress as 192.0.2.10, no matter which node runs them. Everything else takes the normal path. The partner allowlists one address and forgets about your pod churn.

Watch for the sharp edges:

• The gateway node is a chokepoint. All matched egress funnels through it — size it for the throughput, and treat its IP as infrastructure you don't recycle.
• It's a SPOF unless you plan for it. A single egressIP pins to one node; lose that node and the flow stops. Reserve the egress IP as one you can move, and keep multi-node HA egress in mind if the path is business-critical.
• Only traffic matching destinationCIDRs is redirected — scope it tight. Don't send all 0.0.0.0/0 egress through one node by accident.

Gateway API instead of Ingress¶

Ingress is the legacy API. It only ever did host- and path-based HTTP routing, and anything beyond that — header routing, rewrites, mirroring — meant vendor-specific annotations that didn't port between controllers. Gateway API replaces it with a role-oriented, portable model: traffic splitting, header-based routing, URL redirects and rewrites, request mirroring, and TLS termination and passthrough — all first-class fields, no annotation soup.⁵

Cilium ships a built-in Gateway API implementation (v1.3.0 in 1.18), so you can do north-south routing without a separate ingress controller. Enable it at install with --set gatewayAPI.enabled=true, then use the same Gateway and HTTPRoute objects I covered in the Kong on Kubernetes post — Cilium becomes the GatewayClass.

For most teams the split is: Cilium for east-west (pod-to-pod policy and load balancing) and a dedicated gateway like Kong for north-south when you need rich API-management plugins (auth, rate limiting, transformations). Cilium's Gateway API is plenty for straightforward HTTP routing.

Cluster Mesh: one identity model across many clusters¶

Everything so far stops at the cluster boundary. Cluster Mesh erases that boundary — it connects multiple Cilium clusters into one fabric with cross-cluster service discovery, load balancing, and policy that uses the same label-identity model you already wrote.¹¹

The prerequisites are strict, and skipping them is the usual reason a mesh won't form:

• Non-overlapping PodCIDRs across every cluster. Overlap and routing is ambiguous — it simply won't work.
• A unique cluster name and numeric ID (1–255) per cluster, set at install (cluster.name, cluster.id).
• IP reachability between cluster nodes — flat network or a tunnel. Nodes have to be able to route to each other.

Connect two clusters with the CLI — it handles the clustermesh-apiserver and the mTLS cert exchange:

cilium clustermesh enable  --context $CLUSTER1
cilium clustermesh enable  --context $CLUSTER2
cilium clustermesh connect --context $CLUSTER1 --destination-context $CLUSTER2
cilium clustermesh status  --context $CLUSTER1 --wait

Global services: the same Service in every cluster¶

This is the payoff. Deploy the same Service — identical name and namespace — in each meshed cluster, mark it global, and Cilium merges their backends into one virtual service. A pod that calls it gets load-balanced across pods in every cluster, and if one cluster's backends disappear, traffic fails over to another automatically.

Define it in cluster A:

apiVersion: v1
kind: Service
metadata:
  name: payment            # same name…
  namespace: prod          # …same namespace in both clusters
  annotations:
    service.cilium.io/global: "true"
    service.cilium.io/affinity: "local"   # prefer local backends, spill cross-cluster only on failure
spec:
  selector:
    app: payment
  ports:
    - port: 8080

Define the exact same Service in cluster B (same name, same namespace, same global annotation). That matching identity is what tells Cilium "these two are one service" — it stitches cluster A's payment pods and cluster B's payment pods into a single endpoint pool.

Now the caller doesn't change at all. A pod in cluster B that has no local payment pods still just calls the normal in-cluster name and reaches pods in cluster A:

# from any pod in cluster B — standard cluster DNS, no special address
kubectl exec -it deploy/api -n prod -- curl http://payment.prod.svc.cluster.local:8080/health
# → load-balanced to a payment pod in whichever cluster has healthy backends

That's the whole point the user usually misses: the consumer keeps using payment.prod.svc.cluster.local. Cluster Mesh makes that name resolve to pods across clusters — no new endpoint, no gateway, no DNS change.

affinity: local keeps it sane. Without it a request can hop the WAN to a remote cluster even when a healthy local backend exists. With it, traffic stays in-cluster and only spills over when local endpoints are gone — failover without paying cross-cluster latency on every call. Two more knobs round it out:

Policy that spans clusters¶

Because identities are global across the mesh, a CiliumNetworkPolicy can match workloads in another cluster. Scope it with the reserved cluster label when you want to allow only a specific one:

  ingress:
    - fromEndpoints:
        - matchLabels:
            app: api
            io.cilium.k8s.policy.cluster: cluster-1   # only api from cluster-1

So payment in cluster-2 accepts api traffic from cluster-1 and nothing else — zero-trust segmentation that doesn't care which cluster a workload lives in. The mental model stays identical to single-cluster policy; only the blast radius grows.

Service mesh, without the sidecars¶

A service mesh is a dedicated layer for service-to-service traffic: a data plane (proxies moving the bytes) and a control plane (the brain configuring them). It buys you mTLS, traffic control, and observability without touching application code. The question in 2026 isn't whether — it's how much proxy you're willing to run.

The sidecar model (classic Istio). Every pod gets its own Envoy proxy; all inbound and outbound traffic detours through it; a control plane (istiod — the old Pilot, Citadel, and Galley merged into one) configures every proxy. It works, but you pay a proxy container per pod — memory, an extra hop of latency, and a lifecycle to upgrade.

The ambient / sidecarless model. No per-pod proxy. A per-node proxy (Istio's ztunnel, run as a DaemonSet) transparently intercepts pod traffic and provides an L4 mTLS secure overlay, tunneling via HBONE (HTTP-Based Overlay Network) — multiple TCP streams multiplexed over a single mTLS connection. L7 features come from a separate waypoint proxy only when a workload actually needs them.⁹

Cilium's angle: you've already got most of a mesh. Because eBPF sees every packet at the node, Cilium delivers identity-based mTLS, L7 policy, and load balancing without a sidecar — the same sidecarless idea, native to the CNI. The L7 HTTP policy and the free Hubble flow visibility earlier in this post are the mesh data plane; there was never a proxy in your pods.

When a full mesh earns its keep — and when it doesn't:

The honest default: get kube-proxy replacement, identity policy, L7 rules, and Hubble working first. That's a sidecarless mesh for free. Only add a full mesh (Cilium's, or Istio ambient) when you actually hit the left column.

Tetragon: runtime security from the same kernel hooks¶

Once eBPF is in the kernel, you get more than networking. Tetragon (Cilium's sibling project, 1.4 as of February 2026) uses the same machinery to watch the full execve chain — process, namespace, pod metadata — and can kill a process before its syscall completes.¹²

helm install tetragon cilium/tetragon --namespace kube-system

A TracingPolicy that flags any process opening /etc/shadow:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: watch-sensitive-files
spec:
  kprobes:
    - call: "security_file_permission"
      syscall: false
      args:
        - index: 0
          type: "file"
      selectors:
        - matchArgs:
            - index: 0
              operator: "Equal"
              values:
                - "/etc/shadow"

This is optional — you don't need Tetragon to get the networking wins. But it's the natural next layer when "who connected to what" (Hubble) needs to become "what did that process actually do" (Tetragon).

When not to, and the gotchas¶

Cilium isn't free of sharp edges:

• Kernel version is a hard gate. 5.10+ for the features here. Old enterprise distros will fight you — check uname -r on every node first.
• Managed clusters may pin the CNI. On GKE Dataplane V2 you already have Cilium but don't control its config; on others you may not be able to swap the CNI at all.
• kube-proxy-less needs the API server address at install (k8sServiceHost). Get it wrong and Cilium can't bootstrap.
• L7 policy routes through an in-kernel Envoy, which adds a small per-request cost. Worth it for segmentation; don't blanket-apply it to every Service.
• Default-deny breaks things loudly. Roll it out namespace by namespace with Hubble open, not cluster-wide on a Friday.

Summary¶

The eBPF networking stack, in the order you should turn it on:

• Know the three networks — host, pod overlay (PodCIDR), service — and pick a routing mode: encapsulating for any network, native (flat L2 or BGP) to shed the ~50-byte overhead.
• Replace kube-proxy: install with kubeProxyReplacement=true; verify KubeProxyReplacement: True and that the kube-proxy DaemonSet is gone.
• Prove it: cilium status and cilium connectivity test before anything else.
• Turn on Hubble: hubble observe --verdict DROPPED --follow is your new first debugging move.
• Apply default-deny per namespace, then allow-list with CiliumNetworkPolicy bound to labels, not IPs.
• Use L7 rules where it counts — restrict HTTP method/path and DNS egress on sensitive services.
• Egress Gateway when an external system needs a stable source IP — route those pods through a fixed gateway node; scope destinationCIDRs tight.
• Gateway API for north-south if you don't need full API-management; pair with Kong when you do.
• Cluster Mesh to span clusters — non-overlapping PodCIDRs and unique cluster IDs first; global Services with affinity: local for failover without WAN-latency on every call.
• Skip the sidecars: kube-proxy replacement + identity policy + L7 rules + Hubble is a sidecarless mesh. Add a full mesh (Cilium or Istio ambient) only when service count and multi-cluster needs demand it.
• Add Tetragon when you need runtime process visibility and enforcement from the same eBPF layer.

Move networking into the kernel and three things you used to run separately — load balancing, policy, observability — collapse into one layer with no sidecars and no IP churn. The CNI stops being the thing you fight and becomes the thing you forget about.

Questions or discussion? Connect on LinkedIn, X or reach out via email.

Discussion

Have thoughts on this post? Share them below — questions, corrections, or your own experience are all welcome.