Kubernetes Upgrades: Version Skew, Component Order, and Safe Rollout

A Kubernetes upgrade is not a single binary replacement. The API server, etcd, controller manager, scheduler, kubelet, kube-proxy, client tools, network and storage plugins, admission webhooks, and custom controllers have separate compatibility boundaries. An upgrade can appear healthy while mixed component versions are temporarily unsupported or a webhook rejects a newly served API representation.

Treat each minor-version change as a staged compatibility migration with explicit evidence gates. The sequence and supported version skew depend on the Kubernetes release, deployment tool, and managed-service provider. This guide summarizes the upstream rules and a safe operating model; it is not a substitute for the provider’s upgrade calendar, control-plane workflow, or version-specific runbook.

Map the compatibility boundaries first

Kubernetes versions use major, minor, and patch components. Patch updates within a supported minor release generally include fixes without changing the minor API compatibility window. A minor upgrade introduces a new release branch and can remove deprecated API versions or change component behavior. The Kubernetes project maintains support for a limited set of recent minor branches, so “it still runs” does not mean a release is still receiving fixes.

The version-skew policy limits how far apart components may be. For example, in an upstream 1.36-to-1.37 transition, HA kube-apiserver instances may temporarily span one minor version, while kubelet and kube-proxy must not be newer than the API servers they contact. Control-plane controllers and schedulers must not be newer than those API servers and generally may be one minor older. kubectl has its own one-minor compatibility window. During mixed API server versions, a component that can reach either server must satisfy the tighter boundary imposed by the older server.

Do not encode a remembered skew table as a forever rule. The exact policy is maintained by Kubernetes and can change; deployment tools and managed services may be stricter. Check every control-plane and node component against the target release before rolling any part of the cluster. Include network plugins, CSI drivers, device plugins, admission webhooks, autoscalers, monitoring agents, operators, and provider add-ons in the inventory.

Build an upgrade inventory and a go/no-go plan

Before selecting a target version, record the current and target Kubernetes minor and patch versions, distribution/provider, upgrade method, control-plane topology, etcd mode, node pools, and installed extensions. For each extension, note its supported control-plane range, CRD versions, image digests, upgrade order, and rollback path. “Managed Kubernetes” transfers some control-plane operations to the provider; it does not guarantee that workload operators or every installed add-on is compatible.

Review the target release notes and deprecated API guide. Search stored manifests and live objects for removed API versions, and test admission webhooks against the object versions the target API server will send. A webhook that only serves an older version can block creates or updates during an otherwise successful control-plane upgrade. CRD conversion webhooks and storage-version transitions need their own plan; upgrading the API server does not automatically rewrite custom-resource data into a new storage version.

Capture a baseline before maintenance: node readiness, API latency and error rate, workload SLOs, pending Pods, eviction and disruption events, control-plane component health, etcd health and free disk, backup freshness, storage attach behavior, and network policy. Validate capacity to drain nodes without violating application availability. PDBs can limit voluntary evictions, but they do not create replicas, guarantee traffic health, or prevent involuntary failures. Resolve unhealthy replicas and blocked disruptions before beginning.

Agree on stop conditions before the change: API error increase, unavailable replicas, degraded etcd health, a failing admission path, node networking/storage failures, or a drain that cannot proceed within its window. Define who can stop the rollout, what the recovery action is, and how long the team will wait before moving to the next control-plane or worker pool. An operator who starts with no decision gates is likely to improvise under pressure.

Respect the component upgrade order

For self-managed kubeadm clusters, upstream guidance upgrades a primary control-plane node first, then additional control-plane nodes, then worker nodes. The control plane must not skip minor API-server versions. For example, move through each intervening minor release rather than upgrading an older cluster directly to a much newer API server. The exact prerequisites and command syntax are release-specific; kubeadm upgrade plan helps identify compatible plans but does not replace release-note review or workload backups.

The version-skew policy means the API server is upgraded before the controller manager, scheduler, and kubelet components that depend on it. In an HA deployment, while API servers are mixed, components must remain compatible with every API server they may contact. Upgrade the control plane using the provider’s documented sequence, wait for each API server to be healthy and serving, then proceed with dependent control-plane components as directed by the tool. Do not manually change static Pod manifests behind kubeadm or a cloud provider’s orchestration layer.

Treat etcd as its own compatibility and recovery boundary. For a manual control-plane upgrade, upstream’s high-level sequence places all etcd instances before the API servers; for kubeadm, follow the matching kubeadm procedure, which can manage the etcd static-Pod upgrade as part of its operation. In either case, take and verify an etcd backup first, confirm the target etcd version is supported by the Kubernetes release, and observe quorum and API availability during the change. Do not independently upgrade etcd in a provider-managed cluster unless the provider explicitly assigns that operation to you.

Nodes require a separate rollout. A minor-version kubelet upgrade requires draining that node first; upstream documentation does not support an in-place minor kubelet upgrade while Pods remain on it. Many managed services accomplish this by creating replacement nodes, shifting workloads, and retiring old nodes. That blue-green node replacement is often easier to reverse than changing a running operating system package, but it consumes additional subnet IPs, Pod capacity, and storage or license capacity during overlap.

For a kubeadm-operated cluster, commands may resemble:

# Set this to a currently supported target patch before running.
TARGET_VERSION='v1.37.<supported-patch>'
kubeadm upgrade plan "$TARGET_VERSION"
kubeadm upgrade apply "$TARGET_VERSION"

# Drain and upgrade one worker at a time using the release-specific procedure.
kubectl drain worker-01 --ignore-daemonsets
# Upgrade kubelet packages/configuration through the OS package workflow.
kubectl uncordon worker-01

Replace <supported-patch> in the example with an actual supported patch version from the current release and package channel before running it. Do not run kubeadm commands against a cluster whose provider owns the control-plane lifecycle. For managed clusters, use the vendor’s upgrade operation and verify how it sequences control plane, node pools, networking, DNS, proxy, and CSI add-ons.

Drain with application health in view

Draining is an application availability operation, not just a node maintenance command. Check that replicas are distributed across failure domains, that replacement Pods have schedulable capacity, and that storage can detach and reattach where needed. Stateful workloads may be constrained by volume topology or attachment limits. DaemonSet Pods are not evicted like ordinary Pods, and kubectl drain commonly requires --ignore-daemonsets so it can complete while leaving them for node deletion or restart.

Respect PDBs. If a drain is blocked, identify which budget and selector apply, compare disruptionsAllowed with healthy and desired replica counts, and fix the reason there is no disruption allowance. Do not use --disable-eviction as a convenience to bypass PDB checks. If an exception is necessary for an emergency, it must follow the service’s explicit incident policy and be accompanied by a restoration plan.

Drain one canary worker, observe replacement placement and application health, verify CNI and CSI behavior, then return it to service. Continue one node or a small bounded batch at a time, leaving enough spare capacity for workloads to recover. A surge setting that works in a tiny test cluster can exhaust cloud IPs, Pod CIDR blocks, or storage attachments at production scale. Monitor node readiness, pending Pods, PDBs, rollout status, service latency, and volume events continuously.

Define recovery without assuming downgrade is a rollback

An in-place Kubernetes minor-version downgrade is not a normal rollback plan. API storage, etcd, CRDs, and control-plane behavior may have changed; putting an older API server binary back can leave it unable to interpret persisted state or a partially migrated object. Before the change, establish the provider-supported restore procedure, verify an etcd backup or managed control-plane recovery point where applicable, and separately back up application databases and persistent data. A control-plane snapshot is not a backup of application state.

Kubeadm may write local backups of manifests and etcd data for an upgrade, but their presence does not guarantee that an arbitrary partial upgrade can safely be rolled back. Confirm where the backup exists, how to restore it, which members or nodes it covers, and whether the target release’s supported procedure can use it. For managed services, determine whether the provider offers a control-plane restore, only node-pool rollback, or no in-place downgrade at all.

Prefer forward recovery for a limited failure when the control plane is healthy: stop further node batches, restore a compatible add-on or webhook, correct the failing manifest, and resume only after gates pass. If the control plane or persisted data is damaged, follow the tested restore plan rather than inventing a downgrade from package versions. Record the decision and evidence because “rollback” may mean reverting an application deployment, replacing nodes with an earlier image, restoring etcd, or rebuilding a cluster, and those are not equivalent actions.

Validate each layer after upgrade

After each phase, compare observed versions to the expected matrix and verify the cluster can still serve and reconcile workloads:

kubectl version
kubectl get nodes -o wide
kubectl get --raw='/readyz?verbose'
kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded
kubectl get events -A --sort-by=.lastTimestamp

The field selector example is a convenience, not a complete health report: some non-Running Pods are expected, and API servers may restrict supported field selectors. Inspect Deployment, StatefulSet, DaemonSet, Job, and PDB status with workload context. Check API server readiness separately from external load balancers; validate DNS, service routing, network policy, ingress/gateway, storage provisioning and attach, autoscaling, admission, custom-resource conversion, and observability. A node Ready condition proves only a portion of that system is functioning.

Compare metrics to the pre-upgrade baseline and watch long enough to cover representative workload and control-plane cycles. Confirm deprecated API alerts, controller queue depth, API 429/5xx rates, webhook latency and failures, node pressure, and volume events. Verify that all node pools eventually return to the intended target versions and that no component remains at the edge of its allowed skew, which can make the next upgrade impossible.

Production acceptance checklist

The exact target patch and minor releases are supported by the distribution and every required extension.
Current version skew is inventoried for API servers, kubelet, kube-proxy, client tools, and control-plane controllers.
Removed/deprecated APIs, webhook compatibility, CRD conversion/storage versions, and add-on matrices are tested before control-plane mutation.
etcd/provider recovery and application-data backups are tested separately; no unsupported in-place downgrade is assumed.
Capacity includes PDB-respecting drain, node replacement overlap, IP addresses, storage attachments, and autoscaler limits.
The provider-appropriate upgrade order is followed, with canary, phase gates, stop conditions, and owners defined.
Each node batch is validated for readiness, CNI, CSI, DNS, workload health, and service SLOs before the next begins.
All components converge to a supported skew and the next maintenance window remains feasible.

Production-grade upgrades are controlled migrations through a temporary mixed-version system. The version-skew policy defines which combinations are supported; release notes, provider compatibility, workload capacity, and recovery procedures determine whether a particular rollout is safe. Move one boundary at a time, preserve a usable recovery path, and require application evidence before declaring the cluster upgraded.

Related:

Sources: