Kubernetes Static Pods: Kubelet Bootstrap, Mirrors, and Recovery

Kubernetes static Pods are supervised directly by the kubelet on one node. Their desired state comes from a local manifest or a configured web source, not from an object stored in the Kubernetes API. That distinction makes them useful for starting control-plane components before the API server is available, but it also changes how operators must deploy, inspect, and recover them.

The API server can expose a mirror Pod for visibility, but that mirror is not the source of truth. Editing or deleting it with kubectl does not change the locally managed workload. The node’s kubelet configuration and manifest source do. A reliable runbook therefore begins on the node, identifies the configured source, and treats API-visible mirror objects as observations rather than controls.

Where static Pods fit

The kubelet can start a static Pod without asking the scheduler or API server to create it. Kubernetes distributions use this bootstrap property for control-plane components such as the API server, scheduler, controller manager, and sometimes etcd. In a kubeadm-managed cluster, those manifests are normally written under /etc/kubernetes/manifests on each control-plane node.

Static Pods are not a general substitute for Deployments or DaemonSets. They are tied to one kubelet and do not receive control-plane rollout, rollback, or scaling behavior. A node agent that should be placed and updated across a cluster is usually a DaemonSet: the control plane can coordinate its desired copy on each eligible node. Use static Pods when local supervision before or independently of API availability is an explicit requirement, not merely because a process should run on a node.

This creates a clear tradeoff. A static Pod can help bootstrap the service that makes the API available, but its lifecycle and configuration are now part of node operations. Cluster API availability, admission policy, and normal workload controllers cannot repair a bad local manifest while the kubelet is trying to start it. Protect the manifest path, keep changes reviewable, and maintain a recovery route that does not depend on the component being changed.

How the kubelet receives desired state

For filesystem-hosted static Pods, staticPodPath in the kubelet configuration points to a directory or a single manifest file. The kubelet periodically checks the configured source and reconciles the Pods it finds there. The kubelet configuration API also supports a staticPodURL source; that is a separate delivery design with network, availability, and integrity dependencies that must be planned deliberately.

On a kubeadm node, inspect the effective kubelet configuration and the distribution’s service arguments rather than assuming a path from memory. The legacy --pod-manifest-path flag is deprecated in current Kubernetes documentation; the configuration-file field is the maintained interface. Do not add competing sources or alter how the service is launched without understanding how the package or bootstrap tool generates the effective configuration.

The directory scan has a dangerous operational edge: the kubelet ignores files whose names begin with a dot, but it does not filter the remaining files by .yaml or .json extension. A copied backup such as kube-apiserver.yaml.backup can be parsed as another manifest. Two files that define the same Pod name have undefined behavior, and the stale copy may unexpectedly take effect. Keep backups, editor swap files, and generated fragments outside the watched path. Use a separate protected backup directory and restore a reviewed manifest into the active path only when ready.

For a filesystem source, a safe change sequence is:

Identify the active kubelet configuration, manifest path, cluster bootstrap owner, and the exact node being changed.
Save a timestamped copy outside the watched directory and record its checksum and current contents.
Validate the edited Pod manifest and its referenced local files, images, ports, mounts, and security context before installation.
Place only the intended manifest in the watched path using the cluster’s supported deployment or configuration-management process.
Observe kubelet and container-runtime state on that node, then verify control-plane health from an independent vantage point.
Roll out to another control-plane node only after the first node has converged and the cluster has regained its expected health.

This is an operational outline, not an instruction to hand-edit kubeadm-owned files during an upgrade. When kubeadm or another cluster manager owns component manifests, use its documented configuration and upgrade workflow so the local file agrees with the declared cluster configuration. An out-of-band edit can be overwritten by a later reconciliation or create drift that is difficult to diagnose.

Mirror Pods are visibility, not ownership

When the API server is reachable, the kubelet attempts to create a mirror Pod for each static Pod. The mirror appears in the API, commonly in kube-system for control-plane workloads, and is annotated with kubernetes.io/config.mirror. Its name includes the node hostname. This representation lets operators list and inspect node-local Pods through familiar API tooling, but the actual workload remains owned by the kubelet.

Deleting the mirror with kubectl delete pod does not stop the static Pod. The kubelet still sees the local manifest and will recreate the mirror. Likewise, changing fields on the mirror does not edit the source manifest. Use the API object to identify the node and gather metadata, then inspect the local source and runtime when you need to understand or change actual desired state.

Static Pod specs also have important API dependencies that are unavailable by design. They cannot reference API objects such as ConfigMaps, Secrets, or ServiceAccounts, and they do not support ephemeral containers. Configuration needed at bootstrap must be available through an appropriate node-local mechanism or baked into another supported delivery path. Do not assume that a successful kubectl get pod means the Pod was admitted and configured like an ordinary API-created Pod.

Diagnose the node, not only the mirror

Begin with the API view to find the node and recent state, then switch to the host that runs the kubelet. If the API server is down, the first step may be a direct console or out-of-band session because API-based inspection is impossible precisely when a bootstrap component is failing.

kubectl -n kube-system get pods -o wide
kubectl -n kube-system get pod <mirror-pod> -o yaml

Confirm that the object carries the mirror annotation, note its node and events, and avoid treating its deletion as a repair. On the node, inspect kubelet configuration and service logs, then ask the Container Runtime Interface for the actual container state:

sudo systemctl status kubelet
sudo journalctl -u kubelet --since "30 minutes ago"
sudo crictl pods
sudo crictl ps -a
sudo crictl logs <container-id>

Use the runtime endpoint configured for the host if crictl cannot connect; do not infer a container failure from a client pointed at the wrong socket. Match the Pod name and container attempt to the manifest, and correlate kubelet timestamps with runtime events. Look for YAML parse errors, an unreadable or incorrectly configured path, image-pull failures, invalid host mounts, port conflicts, resource pressure, certificate or key problems, and dependencies that are unavailable during boot.

When the manifest was recently changed, compare the active file with the last known-good copy outside the watched directory. Check whether a second non-hidden file defines the same Pod name. Verify the expected staticPodPath in the effective kubelet configuration; a correct manifest in an unwatched directory has no effect. If the kubelet itself is unhealthy, its system service, configuration, node filesystem, and runtime are part of the incident, not just the Pod definition.

For kubeadm control-plane components, review the component-specific kubeadm configuration and the generated manifest as a pair. A manual manifest change may temporarily make the process run, yet later kubeadm upgrade or reconfiguration can replace it. Before reconfiguring a control-plane node, back up the relevant Kubernetes configuration and follow the documented procedure for the installed kubeadm version. In highly available clusters, work one control-plane node at a time and verify API reachability and quorum-sensitive dependencies before proceeding.

Recovery and rollout criteria

If a static Pod is stuck, first preserve evidence: the current manifest, kubelet logs, runtime state, relevant certificates or configuration metadata, and timestamps. Avoid deleting mirror Pods or repeatedly restarting kubelet as a generic first response. A restart may trigger a rescan, but it does not correct a malformed manifest or unavailable dependency and can briefly interrupt unrelated node work.

If a reviewed rollback is needed, restore the last known-good manifest through the same managed process that owns the node configuration. Keep the rollback copy outside the watched directory, verify its checksum and expected component version, then observe whether the kubelet recreates a healthy container. Confirm readiness through the service’s normal health checks and cluster-level signals; a Running container alone does not establish API service health or etcd quorum.

For planned updates, define acceptance criteria before touching a node: the expected mirror and node identity, container image digest or version, component readiness, API request success, and any required etcd health or membership checks. In an HA control plane, keep enough healthy members to preserve availability throughout the sequence. On a single-control-plane cluster, expect API access to be interrupted when replacing the API server static Pod and arrange independent node access and a maintenance window.

Record who owns each manifest, how it is rendered, which source-of-truth configuration generates it, and how to restore the previous version. Alert on sustained kubelet errors, repeated container restarts, missing mirror Pods when the API is healthy, and degraded control-plane health. A missing mirror by itself can also mean the API server is unavailable, so correlate API reachability with host-level evidence instead of paging on a single signal.

Production checklist

Use static Pods only where kubelet-local supervision or pre-API bootstrap is required; choose a controller for ordinary cluster workloads.
Confirm the effective staticPodPath or staticPodURL, kubelet version, runtime endpoint, and configuration owner on every relevant node.
Keep backups and temporary files outside the watched manifest directory; the kubelet does not filter by filename extension.
Treat mirror Pods as read-only operational visibility, not as the authoritative workload configuration.
Do not depend on API objects such as Secrets, ConfigMaps, or ServiceAccounts from a static Pod manifest.
Change kubeadm-managed control-plane manifests through the documented kubeadm workflow and preserve the matching source configuration.
Roll out across HA control-plane nodes serially, checking readiness and quorum between nodes.
Maintain out-of-band node access, a verified last-known-good manifest, and independent service-level health checks.

Static Pods trade API-managed lifecycle features for node-local bootstrap control. They are dependable when that boundary is explicit: the kubelet source is known, manifest directories are kept clean, mirrors are not mistaken for owners, and node-level recovery has been tested before an API outage makes ordinary Kubernetes tooling unavailable.

Related:

Sources: