Fixing a Kubernetes Node Stuck NotReady
kubectl get nodes shows a node stuck in NotReady state, and pods are being evicted from it. Here's how to check kubelet, container runtime, and network plugin health in the right order.
A node showing NotReady means the control plane hasn’t received a recent healthy status report from that node’s kubelet — the fix depends on identifying which specific layer (kubelet itself, the container runtime, or networking) actually stopped reporting healthy.
Step 1: check node conditions for the specific reported reason
kubectl describe node <node-name>The Conditions section frequently states the actual problem directly — MemoryPressure, DiskPressure, or a generic heartbeat timeout each point at a different next step.
Step 2: check kubelet’s own status on the affected node
ssh <node>
systemctl status kubelet
journalctl -u kubelet -n 100If kubelet itself has crashed or isn’t running, nothing else matters until it’s restarted — check its logs for the specific reason it stopped.
Step 3: check container runtime health
systemctl status containerd # or crio, depending on your runtime
crictl infokubelet depends entirely on a healthy container runtime underneath it — a crashed or unresponsive containerd/CRI-O produces a NotReady node even with kubelet itself technically running.
Step 4: check disk pressure specifically
df -hA node that’s run out of disk space (frequently from accumulated container images or logs — see fixing Docker disk space issues for the underlying cause) reports DiskPressure, which kubelet treats as reason enough to stop accepting new pods and mark itself unhealthy.
Step 5: check CNI network plugin health
kubectl logs -n kube-system -l k8s-app=calico-node --field-selector spec.nodeName=<node-name>(Substitute your actual CNI plugin’s namespace/label selector.) A broken network plugin can prevent kubelet from reporting healthy even when the node’s basic compute resources are fine.
Step 6: check node resource pressure generally
kubectl top node <node-name>Confirm actual CPU/memory pressure isn’t simply overwhelming the node — a genuinely overloaded node reporting NotReady may need workload rebalancing (or a Pod Disruption Budget-aware drain) rather than a service restart.
Step 7: restart kubelet after resolving the underlying cause
systemctl restart kubeletRestarting kubelet without first fixing the actual root cause (a full disk, a crashed runtime) just produces the same NotReady state again shortly afterward.
Step 8: cordon and drain the node if it needs deeper investigation or replacement
kubectl cordon <node-name>
kubectl drain <node-name> --ignore-daemonsetsCordoning prevents new pods from scheduling onto a node you’re actively troubleshooting; draining safely evicts existing pods elsewhere first, if the node needs to come out of service for extended repair or replacement.
Why checking conditions before touching anything saves the most time
kubectl describe node’s Conditions section names the actual reported problem in most cases — disk pressure, memory pressure, or a specific heartbeat failure — which tells you immediately whether to look at storage, the container runtime, or networking first, rather than checking all three exhaustively before finding the one that’s actually broken.