Skip to content
SRE & DevOpsHow-To Published Updated 5 min readViews unavailable

How to Set Up Centralized Logging for Kubernetes with the EFK Stack

Build a production-minded Kubernetes logging pipeline with Fluent Bit, Elasticsearch, Kibana, secure transport, retention, and failure controls.

Centralized logging pulls every pod’s logs into one searchable place — this walks through the EFK stack (Elasticsearch, Fluent Bit, Kibana), a common alternative to the original ELK stack that swaps in the lighter-weight Fluent Bit for log collection.

Step 1: choose a supported Elasticsearch deployment

helm repo add elastic https://helm.elastic.co
helm repo update
# Install ECK with a reviewed, explicitly pinned chart version:
helm upgrade --install elastic-operator elastic/eck-operator \
  --namespace elastic-system --create-namespace \
  --version CHART_VERSION

Elasticsearch is the storage and search engine underneath everything else in this stack. Elastic now documents the Elastic Cloud on Kubernetes (ECK) operator as the Kubernetes-native deployment path. The operator installation alone does not create an Elasticsearch cluster: define an Elasticsearch custom resource with reviewed node roles, persistent volumes, resource limits, availability zones, and a supported version. Do not copy a single-node development example into production.

Keep the generated TLS and authentication controls enabled. The actual service name, certificate secret, user secret, and endpoint depend on the name of your ECK resource; capture those values explicitly before configuring the shipper.

Step 2: deploy Fluent Bit as a DaemonSet

helm repo add fluent https://fluent.github.io/helm-charts
helm repo update
helm install fluent-bit fluent/fluent-bit -n logging

Running as a DaemonSet means exactly one Fluent Bit instance runs per node, each responsible for collecting logs from every pod on that specific node — the standard pattern for cluster-wide log collection.

Step 3: configure Fluent Bit to find and parse container logs

[INPUT]
    Name              tail
    Path              /var/log/containers/*.log
    Parser            cri
    Tag               kube.*

[OUTPUT]
    Name  es
    Host  ELASTICSEARCH_SERVICE.logging.svc
    Port  9200
    TLS   On
    Index kubernetes-cluster

This tails the common kubelet symlink path and ships parsed entries to Elasticsearch. Contemporary Kubernetes runtimes write logs in the CRI format, so blindly using the legacy docker parser can split records incorrectly. Verify the runtime and an actual line on your nodes. Configure multiline parsing for stack traces, filesystem-backed buffering, bounded retries, and a dead-letter or alert path so an Elasticsearch outage does not silently discard data or exhaust each node’s disk.

Provide credentials through a Kubernetes Secret mounted or referenced by the chart; never embed passwords in a ConfigMap. Restrict the Elasticsearch user to the indices or data streams the shipper needs, and validate the server certificate rather than disabling TLS verification.

Step 4: enrich logs with Kubernetes metadata

[FILTER]
    Name                kubernetes
    Match                kube.*
    Kube_URL             https://kubernetes.default.svc:443

The Kubernetes filter attaches pod name, namespace, and labels to each log entry — without this, you’d have raw log text with no easy way to filter by which specific workload produced it.

Metadata is useful but can be expensive. Allowlist labels that operators actually query and exclude secrets, authorization headers, personal data, and high-cardinality values such as request IDs from indexed labels. Redaction belongs as close as possible to the producer; a collector-side filter is a second line of defense, not permission to log sensitive material.

Step 5: deploy Kibana for querying and visualization

# Create Kibana with ECK, then use its generated service name:
kubectl port-forward -n logging svc/KIBANA_SERVICE 5601:5601

Step 6: create a data view in Kibana

Kibana → Stack Management → Data Views →
  Create data view → "kubernetes-cluster*"

Step 7: search logs across the entire cluster

Kibana → Discover →
  kubernetes.namespace_name: "production" AND
  kubernetes.labels.app: "myapp"

This is the actual payoff — searching and filtering logs from every relevant pod at once, rather than running kubectl logs against each one individually and manually correlating results.

Step 8: set an index lifecycle policy to control storage growth

Kibana → Stack Management → Index Lifecycle Policies

Logs accumulate quickly at cluster scale — configuring automatic rollover and deletion of old indices prevents Elasticsearch storage from growing unbounded.

Base retention on legal requirements, incident-response needs, ingestion volume, replica count, and recovery objectives. Test lifecycle rules on a noncritical data stream first. A deletion policy is irreversible; snapshots are separate from replicas and should be restored periodically to prove they are usable.

Step 9: build a dashboard for recurring queries

Kibana → Dashboard → Create new dashboard →
  add saved searches and visualizations

Saving common queries (error rates by service, a specific namespace’s recent activity) as a dashboard turns one-off investigation queries into an always-available operational view.

Validate the complete path and its failure modes

Emit a unique test record from a disposable pod, confirm Fluent Bit accepted it, find it in Discover with the expected namespace and pod metadata, and measure ingestion delay. Then temporarily block Elasticsearch in a test environment and verify buffering, retry, recovery, and disk bounds. Alert on rejected records, output retries, buffer growth, Elasticsearch disk watermarks, unavailable shards, and certificate expiry.

Do not confuse centralized logs with a backup or an audit ledger. Application logs can be malformed, sampled, redacted, or deleted by policy. Security events that require tamper evidence need a separately governed destination, stricter access controls, and retention appropriate to that obligation.

Why centralized logging becomes necessary past a certain cluster size

kubectl logs works fine for a handful of pods, but becomes genuinely unusable once a workload runs across dozens of replicas or a cluster hosts many services — there’s no way to search across pods, correlate a request across services, or retain history past a pod’s own lifecycle without something like this stack aggregating logs into one durable, searchable store.

Related:

Sources:

Comments