Kubernetes Container Log Rotation: Limits, kubectl logs, and Durable Retention

Kubernetes makes container stdout and stderr convenient to inspect, but node-local logs have bounded retention. The container runtime captures the streams using the CRI logging format, and the kubelet manages files and rotation. kubectl logs reads through the Kubernetes API from the node; it does not query a durable cluster-wide log store. If a Pod is evicted or its node is lost, its local log files can disappear with it.

This creates two separate retention requirements. Node-level rotation protects disk capacity and limits the local files available for debugging. Cluster-level logging stores and indexes logs independently of the Pod and node lifecycle. Increasing the number of local rotated files does not provide durable retention, and a remote log backend does not eliminate the need to control node disk usage.

Know what the kubelet rotates

The kubelet is responsible for rotating container log files and arranging the logging directory structure, while the container runtime writes the CRI-formatted data. The current Kubernetes logging documentation describes containerLogMaxSize with a default of 10 MiB and containerLogMaxFiles with a default of five files. Kubelet configuration also offers settings for concurrent rotation workers and the interval at which logs are monitored.

These are kubelet configuration settings, not Pod fields. They are generally controlled by the cluster’s node provisioning or managed-service configuration. An application team may not be able to change them directly. Check the actual kubelet configuration through the supported platform control plane and confirm the values on every node pool that runs the workload.

kubectl get pods -n production -l app=catalog -o wide
kubectl describe node worker-01
kubectl get events -A --sort-by=.lastTimestamp

These commands provide cluster-level clues, not a complete inspection of the kubelet’s active log-rotation configuration. Managed Kubernetes providers may expose different APIs or restrict node access. Use the provider’s supported node configuration path, and avoid editing a live node manually if that change will be overwritten by the next replacement or upgrade.

The kubelet can rotate more often than a user expects when a process emits large logs. If a container writes 40 MiB and rotation occurs at 10 MiB, kubectl logs returns at most the latest log file, not the entire 40 MiB history. For a restarted container, kubectl logs --previous can retrieve the prior container instance while that previous log remains available on the node. If multiple containers exist in a Pod, specify the container with -c so the request is unambiguous.

kubectl logs deployment/catalog -n production --tail=200
kubectl logs pod/catalog-6bf5589cdb-k4p9d -n production \
  -c catalog --previous --tail=200

--previous refers to the prior terminated container instance for that Pod, not an unlimited history of every restart. Local files may be cleaned up by kubelet garbage collection or disappear when the Pod is evicted. Capture logs to an external system before they are needed for an incident; do not treat an interactive kubectl logs command as archival storage.

Write structured application logs to stdout and stderr

Kubernetes’ most common logging path is an application writing logs to standard output and standard error. This lets the runtime and kubelet manage the container log lifecycle and makes kubectl logs and node-level collection agents straightforward. Use structured records with timestamps, severity, service identity, and request or trace correlation IDs where appropriate. Keep each record self-contained because separate containers and streams can interleave in a collector.

Avoid writing a log file and then copying the same data to stdout unless a specific tool requires that architecture. Duplicating data can increase node storage and create two sources that diverge during rotation. If a legacy application only writes to a file, configure it to write to /dev/stdout or /dev/stderr when supported, or use a sidecar only when its lifecycle and volume semantics are understood. Kubernetes documentation notes that file-to-stdout streaming can use additional storage.

Do not log unbounded payloads, credentials, or entire request bodies by default. Log volume affects disk, network, and backend costs, and rotation is not a data-retention control. Apply application-level sampling or filtering based on explicit operational requirements, and ensure error logs retain enough information to diagnose failures without exposing sensitive values.

Cluster-level collection is a separate service

Kubernetes does not provide a native durable log storage backend. A cluster-level logging architecture usually runs an agent on each node or in each workload context and forwards records to a separate storage and query system. The agent, backend, network path, buffering, indexing, and retention policy form a separate operational system that must be monitored.

Design collection around the source and failure model. A node agent must read the correct runtime log directory with permissions appropriate to the platform. It should enrich records with namespace, Pod, container, and node metadata. If the backend is unavailable, the agent needs a bounded buffer and explicit overflow behavior; an infinite buffer is not possible on a finite node disk. Monitor dropped-record counters, queue depth, delivery delay, backend acceptance, and disk usage.

A logging backend may keep logs after Pods are deleted, but only if collection occurred before local removal and the backend’s retention policy retains the data. Test with a Pod that writes a known marker, restarts, is deleted, and then has its node drained. Verify that the marker remains queryable after each event and that metadata still identifies the original workload revision.

Logs also consume local ephemeral storage

Container logs can contribute to node disk pressure. A workload with excessive output may fill the node’s filesystem even if its writable layer and emptyDir usage look small. Ephemeral-storage requests and limits, kubelet eviction thresholds, container log rotation, image storage, and system-level logs are related contributors but are accounted for through distinct platform behavior.

Correlate node DiskPressure events with Pod placement, log rates, local ephemeral storage metrics, and rotation configuration. A fixed file-size limit does not make a high-volume application safe: the total number of files multiplied by the number of containers and Pods still has a cost, and other node workloads share storage. Use dashboards to identify both per-container output rate and total node filesystem utilization.

Do not manually delete active runtime log files on a node as a routine cleanup. The kubelet and runtime manage file handles and rotation state; deleting files behind their back can make usage accounting and recovery confusing. If a node is out of disk, follow the provider’s supported node remediation process, preserve evidence, and fix the application or node configuration that produced the pressure.

Validate the retention contract before an incident

Measure representative log volume, size the node-level rotation budget, and select a cluster backend retention period that satisfies operational and regulatory needs. The values should be a deliberate capacity calculation: per-container size and file count, expected container count per node, node disk size, system logs, image storage, and peak output rate. Include burst behavior rather than planning only for a quiet day.

Run a test that emits known records before and after the rotation threshold, restarts a container, and confirms what kubectl logs and --previous return. Separately query the cluster-level backend after Pod deletion and node replacement. Ensure an alert fires when log collection falls behind, when the backend rejects records, and when node disk use approaches the kubelet’s pressure threshold.

Document which system owns each part: the application owns log quality and volume; the kubelet and runtime own local files and rotation; the node agent owns collection and buffering; the backend owns durable retention and querying; the platform team owns node configuration and upgrade behavior. Clear ownership prevents an incident response from mistaking “visible through kubectl” for “safely retained.”

Estimate local log capacity with a per-node model before raising rotation limits. For each workload, multiply peak bytes per second by the expected collection delay, then account for the configured file size and count across the maximum number of containers that can land on the node. Add space for container writable layers, images, system logs, and kubelet metadata, then leave operating headroom for bursts and cleanup lag. A rough formula is useful for finding impossible plans; validate the final result with measured node filesystem usage under a load test.

Monitor bytes written and retained by container, file-rotation frequency, node filesystem free space, eviction events, and log-agent delivery lag. A sudden increase in rotation can indicate a noisy application even when the node disk remains below its threshold. A quiet agent queue combined with missing records may indicate that the agent stopped tailing files or lost its cursor. Alert on both storage pressure and delivery health so the team does not solve one by silently accepting data loss.

Related:

• How to Set Up Centralized Logging for Kubernetes with the EFK Stack
• Kubernetes Local Ephemeral Storage: Accounting, Limits, and DiskPressure

Sources:

• Kubernetes: Logging Architecture
• Kubernetes: System Logs