Fixing OOMKilled Containers with Correct Resource Limits

OOMKilled means a container exceeded its configured memory limit and the kernel’s OOM killer terminated it — distinct from a pod stuck in CrashLoopBackOff for other reasons, this specifically means the memory ceiling itself was too low for what the application actually needs.

Step 1: confirm OOMKilled is actually the termination reason

kubectl describe pod mypod | grep -A3 "Last State"

Last State: Terminated, Reason: OOMKilled confirms this specifically — a different termination reason means a different problem entirely.

Step 2: check the currently configured memory limit

kubectl get pod mypod -o jsonpath='{.spec.containers[0].resources}'

Step 3: measure the application’s actual peak memory usage

kubectl top pod mypod --containers

Watching this over a representative period (including peak load, not just idle) shows what the application genuinely needs — setting limits based on a guess, rather than measured behavior, is the single most common cause of both OOMKills (limit too low) and wasted cluster capacity (limit too generous).

kubectl top is a recent sample, not a peak recorder, and the killed process may already be gone. Use the monitoring system’s container working-set, RSS, cache, restart, and limit series at sufficient resolution around the termination. Correlate with traffic, jobs, deployments, and node memory pressure. Confirm which container failed in a multi-container pod.

Step 4: check for a genuine memory leak versus a legitimately high baseline

kubectl top pod mypod --containers --use-protocol-buffers

Watched over time: memory usage that climbs continuously without ever stabilizing indicates a real leak in the application, which raising the limit only delays rather than fixes; usage that rises to a stable plateau under load is a legitimately higher baseline the limit should simply accommodate.

Step 5: set a limit reflecting genuine peak usage, with reasonable headroom

resources:
  requests:
    memory: "512Mi"
  limits:
    memory: "768Mi"

A limit set right at the observed peak leaves no room for normal variance and will OOMKill again under slightly heavier load — some headroom above measured peak is appropriate, though a limit dramatically higher than actual usage just wastes schedulable capacity on the node.

Step 6: distinguish request from limit deliberately

requests is what the scheduler uses to place the pod on a node with enough capacity; limits is the hard ceiling the kernel enforces. Setting these too close together removes the pod’s ability to use any burst capacity beyond its baseline; setting limits far above requests risks a node overcommitting memory across many pods that all burst simultaneously.

Step 7: check for a language-runtime-specific memory ceiling misconfigured independently of the container limit

# JVM example — must be set BELOW the container's own memory limit
-Xmx512m

For JVM, Node.js, and similar runtimes with their own internal heap limits, a runtime-level memory ceiling set higher than the container’s own limit means the container gets OOMKilled by the kernel before the application runtime’s own memory management ever kicks in — these two settings need to be coordinated, not configured independently.

Heap is not total process memory. Native allocations, threads, stacks, buffers, memory-mapped files, JIT code, and sidecar usage need headroom inside the cgroup limit. Prefer runtime options that understand container limits, and verify the exact supported flags for the deployed runtime version rather than copying a percentage blindly.

Separate container OOM from node-level eviction

An OOMKilled termination generally points to the container cgroup limit, while kubelet eviction events indicate node memory pressure. A node can also invoke the system OOM killer when overcommitted; inspect node kernel logs and Kubernetes events where authorized. Raising one container limit may move the failure to neighboring workloads if node capacity and requests remain unchanged.

Profile suspected leaks with language-native tooling in a controlled environment. Heap dumps can contain credentials and customer data, so encrypt, restrict, and delete them under an explicit retention policy. For cache-heavy applications, determine whether memory is reclaimable and whether the application has a bounded cache configuration.

After changing requests, limits, or code, test realistic concurrency, dataset size, warmup, and long-running behavior. Confirm no new OOM terminations, acceptable garbage-collection pauses, enough node headroom, and correct scheduling. Alert on restarts and approach to limits before customers discover the next kill.

Step 8: verify the fix under realistic load

kubectl top pod mypod --containers

Confirm the container now runs through representative peak load without hitting the new limit, and check kubectl describe pod after some time to confirm no further OOMKilled restarts occurred.

Why measuring actual usage beats guessing at a “reasonable” limit

Resource limits set from assumption rather than measurement are wrong in one of two directions almost every time — too low, causing exactly this OOMKilled cycle, or too generous, quietly wasting cluster capacity that could schedule other workloads. kubectl top over a representative period, covering real peak load rather than just idle behavior, is what turns limit-setting from a guess into a measurement-backed decision.

Related:

Sources: