Fixing OOMKilled Containers with Correct Resource Limits

OOMKilled means a container exceeded its configured memory limit and the kernel’s OOM killer terminated it — distinct from a pod stuck in CrashLoopBackOff for other reasons, this specifically means the memory ceiling itself was too low for what the application actually needs.

Step 1: confirm OOMKilled is actually the termination reason

kubectl describe pod mypod | grep -A3 "Last State"

Last State: Terminated, Reason: OOMKilled confirms this specifically — a different termination reason means a different problem entirely.

Step 2: check the currently configured memory limit

kubectl get pod mypod -o jsonpath='{.spec.containers[0].resources}'

Step 3: measure the application’s actual peak memory usage

kubectl top pod mypod --containers

Watching this over a representative period (including peak load, not just idle) shows what the application genuinely needs — setting limits based on a guess, rather than measured behavior, is the single most common cause of both OOMKills (limit too low) and wasted cluster capacity (limit too generous).

Step 4: check for a genuine memory leak versus a legitimately high baseline

kubectl top pod mypod --containers --use-protocol-buffers

Watched over time: memory usage that climbs continuously without ever stabilizing indicates a real leak in the application, which raising the limit only delays rather than fixes; usage that rises to a stable plateau under load is a legitimately higher baseline the limit should simply accommodate.

Step 5: set a limit reflecting genuine peak usage, with reasonable headroom

resources:
  requests:
    memory: "512Mi"
  limits:
    memory: "768Mi"

A limit set right at the observed peak leaves no room for normal variance and will OOMKill again under slightly heavier load — some headroom above measured peak is appropriate, though a limit dramatically higher than actual usage just wastes schedulable capacity on the node.

Step 6: distinguish request from limit deliberately

requests is what the scheduler uses to place the pod on a node with enough capacity; limits is the hard ceiling the kernel enforces. Setting these too close together removes the pod’s ability to use any burst capacity beyond its baseline; setting limits far above requests risks a node overcommitting memory across many pods that all burst simultaneously.

Step 7: check for a language-runtime-specific memory ceiling misconfigured independently of the container limit

# JVM example — must be set BELOW the container's own memory limit
-Xmx512m

For JVM, Node.js, and similar runtimes with their own internal heap limits, a runtime-level memory ceiling set higher than the container’s own limit means the container gets OOMKilled by the kernel before the application runtime’s own memory management ever kicks in — these two settings need to be coordinated, not configured independently.

Step 8: verify the fix under realistic load

kubectl top pod mypod --containers

Confirm the container now runs through representative peak load without hitting the new limit, and check kubectl describe pod after some time to confirm no further OOMKilled restarts occurred.

Why measuring actual usage beats guessing at a “reasonable” limit

Resource limits set from assumption rather than measurement are wrong in one of two directions almost every time — too low, causing exactly this OOMKilled cycle, or too generous, quietly wasting cluster capacity that could schedule other workloads. kubectl top over a representative period, covering real peak load rather than just idle behavior, is what turns limit-setting from a guess into a measurement-backed decision.