Centralize Log Solution with the Elastic Stack¶

Use case: ship Kubernetes ingress logs off the cluster into a self-managed Elastic Stack on Ubuntu servers, parse them, and keep them searchable for 90 days without blowing the disk budget. The Elastic Stack handles this cleanly, and the 2026 build leans on four current features that older tutorials skip: LogsDB, ILM, data streams, and a Kubernetes-native shipper.

Everything here is runnable. The setup, shipper, pipeline, test, and day-2 scripts live in the elastic-centralized-logging project: a docker-compose cluster you can run on a laptop plus the Ubuntu install scripts for production. Each section links to the file that implements it.

Verified against Elastic Stack 9.3 (June 2026).

The pipeline¶

Centralized logging is a pipeline with a buffer in the middle so a log storm can't knock over Elasticsearch:

 Kubernetes cluster                          Centralized ELK on Ubuntu servers
 ┌──────────────────────────┐                ┌──────────────────────────────────────────┐
 │ ingress-nginx access logs│                │  Logstash x2 (parse + enrich)              │
 │   (one log stream/node)  │                │  Ubuntu, apt                               │
 │           │              │   Beats over    │        │                                  │
 │  Filebeat DaemonSet ─────┼──  Kafka ──────▶│        ▼                                  │
 │  (one pod per node)      │   (buffer)      │  Elasticsearch cluster (Ubuntu, apt)      │
 └──────────────────────────┘                │   3 master / 3 hot / 2 warm / 2 frozen     │
                                             │        │                          │        │
                                             │        ▼                          ▼        │
                                             │   Kibana x2 (Ubuntu) ◀──┘     MinIO / S3    │
                                             └──────────────────────────────────────────┘

Each stage is one component, chosen for what it does best in a 2026 Kubernetes-to-Ubuntu deployment:

What changed and why it matters¶

• LogsDB index mode cuts log storage up to 65% (76% at scale) through index sorting, synthetic _source, and better codecs. It's automatic for logs-* data streams since 9.2.¹²
• ILM, not Curator. Index Lifecycle Management deletes old data and moves it across hot/warm/cold/frozen tiers natively, configured once on the data stream. The older cron-based Curator tool isn't needed.³
• Frozen tier on object storage mounts searchable snapshots from S3/MinIO, pushing 90-day retention to object-store prices instead of SSD.⁴
• Shard rules updated: target 30–50 GB per shard, ≤1000 shards/node, heap ≤31 GB.⁵

Source: ship ingress-nginx logs off Kubernetes¶

ingress-nginx writes one access-log line per request to stdout, so a Filebeat DaemonSet (one pod per node) tails every node's container logs, adds Kubernetes metadata, and forwards them. Make life easy downstream by configuring ingress-nginx to log JSON instead of the default text format:

# ingress-nginx ConfigMap: structured access logs, no brittle grok needed
apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  log-format-escape-json: "true"
  log-format-upstream: >
    {"time":"$time_iso8601","remote_addr":"$remote_addr","method":"$request_method",
    "path":"$uri","status":$status,"bytes":$body_bytes_sent,"request_time":$request_time,
    "upstream_status":"$upstream_status","upstream_time":"$upstream_response_time",
    "host":"$host","user_agent":"$http_user_agent","req_id":"$req_id"}

The Filebeat DaemonSet tails ingress-nginx pods and ships to Logstash (or Kafka). Full manifest: shippers/filebeat-daemonset.yaml. The key bits:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - condition: { equals: { kubernetes.namespace: "ingress-nginx" } }
          config:
            - type: container
              paths: ["/var/log/containers/*${data.kubernetes.container.id}.log"]
output.kafka:                       # or output.logstash for a small setup
  hosts: ["kafka-0.logging:9092"]
  topic: "ingress-logs"

Why a DaemonSet? Filebeat is a lightweight Go binary, and its Kubernetes autodiscover attaches pod, namespace, and label metadata to every line automatically. That's exactly what you need to slice ingress logs by service or namespace later, and it's why a node-level agent beats forwarding raw syslog.

Buffer: Kafka (or Redis) absorbs the spike¶

A deploy or a traffic surge can spike ingress logs 30× in seconds. The buffer takes that hit so Elasticsearch never sees backpressure. Kafka is the scalable choice (durable, replayable, partitioned); Redis is fine for small clusters. Filebeat writes to the buffer, Logstash reads from it at a rate the cluster can sustain.

Logstash on Ubuntu: parse the ingress logs¶

Logstash pulls from the buffer, parses each line into fields, and writes to a data stream. With JSON logs the filter is trivial; a grok fallback handles the default text format. Pipeline: compose/logstash/pipeline/ingress-nginx.conf.

filter {
  if [message] =~ /^\{/ {
    json { source => "message" }            # JSON ingress logs (recommended)
  } else {
    grok {                                   # fallback: default text format
      match => { "message" =>
        '%{IPORHOST:remote_addr} - %{DATA:user} \[%{HTTPDATE:ts}\] "%{WORD:method} %{DATA:path} HTTP/%{NUMBER:httpver}" %{NUMBER:status} %{NUMBER:bytes} "%{DATA:referer}" "%{DATA:agent}" %{NUMBER:req_len} %{NUMBER:request_time} \[%{DATA:upstream}\] %{DATA:upstream_addr} %{DATA:upstream_len} %{DATA:upstream_time} %{DATA:upstream_status} %{DATA:req_id}' }
    }
  }
  mutate { convert => { "status" => "integer"  "bytes" => "integer"  "request_time" => "float" } }
}
output {
  elasticsearch {
    hosts => ["https://es01:9200"]
    data_stream => "true"
    data_stream_dataset => "nginx.ingress"   # -> data stream logs-nginx.ingress-default
  }
}

Routing to a logs-nginx.ingress-* data stream means LogsDB and ILM apply automatically. Install Logstash on Ubuntu with the Elastic apt repo (next section).

Install Elasticsearch, Logstash, Kibana on Ubuntu¶

One apt repo serves all three. Script: ubuntu/install-elastic-ubuntu.sh.

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch \
  | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] \
  https://artifacts.elastic.co/packages/9.x/apt stable main" \
  | sudo tee /etc/apt/sources.list.d/elastic-9.x.list
sudo apt update
sudo apt install -y elasticsearch        # on each ES node
# sudo apt install -y logstash           # on Logstash nodes
# sudo apt install -y kibana             # on Kibana nodes

Each Elasticsearch node gets its tier role in /etc/elasticsearch/elasticsearch.yml:

cluster.name: logging-platform
node.name: es-hot-01
node.roles: [ data_hot, data_content, ingest ]   # warm/cold/frozen nodes set their own
network.host: 0.0.0.0
discovery.seed_hosts: ["es-master-01","es-master-02","es-master-03"]
cluster.initial_master_nodes: ["es-master-01","es-master-02","es-master-03"]

Set heap to ≤31 GB and no more than half of RAM in /etc/elasticsearch/jvm.options.d/heap.options. Open 9200 (API) and 9300 (cluster) between nodes with ufw.

Cluster topology: scalable and HA¶

Separate node roles so each scales on its own. The non-negotiable for HA is at least 2 nodes per data tier (so a replica lives on a different box) and 3 masters for quorum.

LogsDB and the data stream template go on once: setup/40-component-templates.sh and setup/50-index-template.sh.

Retention with ILM¶

ILM walks each data stream through the tiers and deletes it at the end, configured once and applied automatically through the index template:

PUT _ilm/policy/logs-lifecycle
{
  "policy": { "phases": {
    "hot":    { "actions": { "rollover": { "max_primary_shard_size": "50gb", "max_age": "1d" } } },
    "warm":   { "min_age": "7d",  "actions": { "shrink": {"number_of_shards":1}, "forcemerge": {"max_num_segments":1} } },
    "frozen": { "min_age": "30d", "actions": { "searchable_snapshot": {"snapshot_repository":"minio-repo"} } },
    "delete": { "min_age": "90d", "actions": { "delete": {} } }
  } }
}

Script: setup/30-ilm-policy.sh. Inspect any data stream's phase with day2/ilm-explain.sh.

Sizing for ingress volume¶

Size from daily volume and retention, adjusted for LogsDB and replicas:

Hot storage per node = (GB/day × hot_days × (1 + replicas) × 0.35) / (max_disk_pct/100) / hot_nodes

The 0.35 is the LogsDB factor (≈65% savings). Worked example at 50 GB/day of ingress logs, replica=1, 7 days hot, 85% watermark, 3 hot nodes:

(50 × 7 × 2 × 0.35) / 0.85 / 3  ≈ 96 GB per hot node

Without LogsDB that's ~275 GB per node. Keep primary shards 30–50 GB, and add a hot node when ingest pushes shards past 50 GB or a node nears 1000 shards or 85% disk.

Prove it survives a node failure¶

A single hot node means a single copy of today's logs. Test that you actually have redundancy: test/ha-failover-test.sh kills the hot node and asserts the cluster goes yellow, not red, with queries still answered from replicas.

Run the whole flow locally¶

git clone https://github.com/pkhamdee/elastic-centralized-logging
cd elastic-centralized-logging/compose
cp .env.example .env                 # set passwords
docker compose up -d                 # ES x3 + Kibana + Logstash + Redis + MinIO
cd ..
export ES_PASS="$(grep ELASTIC_PASSWORD compose/.env | cut -d= -f2)" ES_CACERT="compose/certs/ca/ca.crt"
./setup/run-all.sh                   # ILM, LogsDB templates, snapshot repo, roles
./test/parse-ingress-test.sh         # feed a sample ingress log, verify it parses into the data stream
./test/ha-failover-test.sh           # prove HA

Day-2 operations¶

Summary¶

The centralized ingress-log platform that scales and stays up:

• Source: ingress-nginx logging JSON, tailed by a Filebeat DaemonSet with Kubernetes metadata.
• Buffer: Kafka (Redis for small) so a log storm can't backpressure Elasticsearch.
• Parse: Logstash on Ubuntu, JSON-first with a grok fallback, writing to a logs-nginx.ingress-* data stream.
• Store: Elasticsearch on Ubuntu (apt), separate node roles, LogsDB on, replica ≥ 1.
• Retain: ILM hot → warm → frozen → delete, frozen on MinIO/S3. No Curator cron.
• HA: 2+ nodes per tier, 3 masters. Prove it with the failover test.
• Size from GB/day, shards 30–50 GB, add nodes (not bigger boxes) at the limits.

Collect, buffer, parse, store, retire. Each stage scales on its own, and every piece is a current Elastic feature you can point at its product docs.

Questions or discussion? Connect on LinkedIn, X or reach out via email.

Discussion

Have thoughts on this post? Share them below — questions, corrections, or your own experience are all welcome.