Diagnosing and Fixing CrashLoopBackOff in Kubernetes

kubectl get pods shows a pod stuck in CrashLoopBackOff — the container starts, exits, and Kubernetes restarts it with an exponentially increasing backoff delay between attempts. This status itself isn’t the problem; it’s Kubernetes correctly reporting that the actual underlying problem keeps recurring.

Step 1: read the pod’s own event history first

kubectl describe pod mypod

The Events section at the bottom frequently states the issue directly — an image pull failure, a failed liveness probe, an OOMKilled event, or a container exiting with a specific, non-zero code.

Step 2: check the container’s actual logs

kubectl logs mypod
kubectl logs mypod --previous

--previous is essential here — once a container has already restarted, plain kubectl logs shows the current (freshly restarted, likely mostly-empty) container’s logs, not the ones from the crash you’re actually trying to diagnose. The application’s own stack trace or error message, from the previous instance’s logs, is usually the most direct path to the actual cause.

Step 3: check for an OOMKill specifically

kubectl describe pod mypod | grep -A3 "Last State"

Last State: Terminated, Reason: OOMKilled means the container exceeded its memory limit and the kernel’s OOM killer terminated it — a resource-configuration problem, not an application bug per se. The fix is either raising the pod’s memory limits, or (if the application is genuinely leaking memory) fixing the leak itself:

resources:
  limits:
    memory: "512Mi"
  requests:
    memory: "256Mi"

Step 4: check for a failing liveness probe killing an otherwise-healthy container

If the container’s own logs look completely normal with no errors, but it still gets killed and restarted, check whether a misconfigured liveness probe is doing this — a probe with too short a timeout, or checking the wrong port/path, kills a perfectly healthy container that just hadn’t finished starting up yet:

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10

initialDelaySeconds set too low relative to the application’s actual startup time is one of the most common self-inflicted causes of CrashLoopBackOff — the container is fine, but Kubernetes concludes it’s unhealthy before it’s finished initializing.

Step 5: check for a configuration or secret that’s missing or wrong

An application that immediately exits (rather than running for a while before crashing) often indicates a startup-time failure — a missing environment variable, an unreachable database connection string, or a ConfigMap/Secret that isn’t mounted where the application expects:

kubectl exec -it mypod -- env
kubectl get configmap myconfig -o yaml

Step 6: check the exit code for a direct clue

kubectl get pod mypod -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'

Exit code 1 usually means a generic application error (check logs); 137 means the process was sent SIGKILL (often an OOMKill, per step 3); 143 means SIGTERM (a graceful shutdown request that the application may not be handling properly).

Do not infer the cause from an exit code alone. Signal-derived codes are conventions, and 137 can result from an explicit kill rather than memory pressure. Correlate terminated state, node events, logs, restart time, and metrics. For multi-container pods, specify -c CONTAINER; otherwise you may inspect a healthy sidecar.

Preserve evidence and debug safely

Record pod YAML, events, previous logs, owner revision, and image digest before changing anything. Events expire, logs rotate, and deleting the pod destroys useful context. If the image lacks a shell or exits too quickly, use kubectl debug with an ephemeral container when policy permits it; do not rebuild production images with ad hoc tools as the first diagnostic step.

Check init containers because failure there prevents the main container from starting. Also inspect startup probes: they defer liveness checks during slow initialization without making an unhealthy process Ready. Readiness failures remove a pod from Service endpoints but do not restart its container.

After correction, watch a full rollout and representative load:

kubectl rollout status deployment/myapp
kubectl get pods -w

Confirm restart counts stop increasing and the customer path works. Preserve the failure signature in the incident record so it can be alerted on rather than rediscovered.

Why CrashLoopBackOff is a symptom, not a diagnosis

Deleting and recreating a pod in CrashLoopBackOff almost never fixes anything, since Kubernetes will simply recreate the exact same failure the moment the new pod starts — the backoff status is Kubernetes correctly identifying that something underneath the pod is broken, and steps 1 through 6 above are what actually identifies which of the several common causes (OOM, a bad probe, a missing config, an application bug) is the real one in your specific case.

Related:

Sources: