Fixing a Kubernetes Node Stuck NotReady
Diagnose a Kubernetes NotReady node through conditions, leases, kubelet, runtime, storage, networking, and safe workload evacuation with evidence intact.
A node showing NotReady means the control plane hasn’t received a recent healthy status report from that node’s kubelet — the fix depends on identifying which specific layer (kubelet itself, the container runtime, or networking) actually stopped reporting healthy.
Step 1: check node conditions for the specific reported reason
kubectl describe node <node-name>The Conditions section frequently states the actual problem directly — MemoryPressure, DiskPressure, or a generic heartbeat timeout each point at a different next step.
Step 2: check kubelet’s own status on the affected node
ssh <node>
systemctl status kubelet
journalctl -u kubelet -n 100If kubelet itself has crashed or isn’t running, nothing else matters until it’s restarted — check its logs for the specific reason it stopped.
Step 3: check container runtime health
systemctl status containerd # or crio, depending on your runtime
crictl infokubelet depends entirely on a healthy container runtime underneath it — a crashed or unresponsive containerd/CRI-O produces a NotReady node even with kubelet itself technically running.
Step 4: check disk pressure specifically
df -hA node that’s run out of disk space (frequently from accumulated container images or logs — see fixing Docker disk space issues for the underlying cause) reports DiskPressure, which kubelet treats as reason enough to stop accepting new pods and mark itself unhealthy.
Step 5: check CNI network plugin health
kubectl logs -n kube-system -l k8s-app=calico-node --field-selector spec.nodeName=<node-name>(Substitute your actual CNI plugin’s namespace/label selector.) A broken network plugin can prevent kubelet from reporting healthy even when the node’s basic compute resources are fine.
Step 6: check node resource pressure generally
kubectl top node <node-name>Confirm actual CPU/memory pressure isn’t simply overwhelming the node — a genuinely overloaded node reporting NotReady may need workload rebalancing (or a Pod Disruption Budget-aware drain) rather than a service restart.
Step 7: restart kubelet after resolving the underlying cause
systemctl restart kubeletRestarting kubelet without first fixing the actual root cause (a full disk, a crashed runtime) just produces the same NotReady state again shortly afterward.
Before restarting, save recent kubelet and runtime logs, node conditions, events, and the last Lease renewal. A restart can erase timing clues. On managed nodes, use the provider-supported diagnostic and replacement workflow; manual service changes may be overwritten or place the instance outside support.
Step 8: cordon and drain the node if it needs deeper investigation or replacement
kubectl cordon <node-name>
kubectl drain <node-name> --ignore-daemonsetsCordoning prevents new pods from scheduling onto a node you’re actively troubleshooting; draining safely evicts existing pods elsewhere first, if the node needs to come out of service for extended repair or replacement.
Distinguish loss of contact from resource pressure
The node controller uses status updates and Lease heartbeats to determine reachability. Ready: Unknown with stale heartbeats suggests the control plane cannot hear from kubelet, which may be host failure, routing, firewall, certificate, clock, or API connectivity. Ready: False with current reports indicates kubelet is communicating but declares itself unhealthy. That distinction changes where to investigate.
Check filesystem bytes and inodes, runtime sockets, DNS, routes, MTU, and CNI pods specifically on the affected node. kubectl top depends on the metrics pipeline and may be unavailable precisely during a node failure, so corroborate with host metrics. Do not assume a Calico label when another CNI is installed.
Drain only after confirming replacement capacity and PodDisruptionBudgets. Local data, unmanaged pods, DaemonSets, and unhealthy workloads require explicit decisions; avoid --force until the ownership and data-loss consequences are understood. If the node is unreachable, eviction may proceed only after controller timeouts, and storage attachments can delay rescheduling.
After repair or replacement, uncordon only when Ready is stable, runtime and CNI health are clean, test pods can start and reach dependencies, and alerts have cleared. Repeated NotReady transitions usually warrant replacement and a root-cause investigation rather than another restart.
Why checking conditions before touching anything saves the most time
kubectl describe node’s Conditions section names the actual reported problem in most cases — disk pressure, memory pressure, or a specific heartbeat failure — which tells you immediately whether to look at storage, the container runtime, or networking first, rather than checking all three exhaustively before finding the one that’s actually broken.
Related:
Sources: