Skip to content
SRE & DevOpsHow-To Published Updated 3 min readViews unavailable

How to Back Up and Restore Kubernetes etcd Without Creating a False Recovery Plan

A recovery-first etcd procedure covering consistent snapshots, encryption keys, revision bumps, restore topology, and proof that Kubernetes reconciles.

An etcd snapshot is necessary for recovering a Kubernetes control plane, but the existence of one .db file does not prove a cluster can be restored. The backup must be consistent, protected with the other secrets the control plane needs, copied off the failed infrastructure, and exercised through a documented restore. A snapshot that nobody has restored is an untested hypothesis.

Define the recovery object before running a command

etcd stores Kubernetes API state: workloads, configuration, RBAC, leases, and Secret objects. It does not contain container images, persistent-volume data, cloud load balancers, node-local files, or external databases. Record separate recovery procedures for those systems and state explicit recovery-point and recovery-time objectives.

For a kubeadm-style static control plane, identify the etcd version, peer topology, certificate paths, data directory, snapshot destination, and whether Kubernetes encrypts API data at rest. Preserve the encryption configuration and its keys separately; restoring encrypted Secret values without the key material leaves ciphertext that the API server cannot decode.

Take a consistent snapshot

Use the etcdctl version compatible with the running etcd release and connect to one healthy member through authenticated TLS. Paths differ by distribution, so inspect the static Pod manifest or vendor documentation rather than copying these examples blindly:

export ETCDCTL_API=3
etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /secure-backup/etcd-2026-08-09.db

Do not copy a live member’s data directory as if it were an application-level snapshot. snapshot save asks etcd for a consistent point-in-time database image. Immediately inspect the result:

etcdutl snapshot status /secure-backup/etcd-2026-08-09.db --write-out=table
sha256sum /secure-backup/etcd-2026-08-09.db

Modern etcd documentation uses etcdutl for offline snapshot inspection and restore. Version-pin the tooling in the runbook because older guides use etcdctl snapshot restore and flags can move between releases.

Protect the complete recovery set

Store the snapshot, checksum, manifest, etcd and Kubernetes versions, CA and serving-certificate plan, encryption configuration, and infrastructure definitions in access-controlled off-cluster storage. Private keys deserve stricter handling than the snapshot itself. Encrypt backups, separate deletion authority from routine cluster credentials, test retention, and alert when scheduled snapshots or copies fail.

Keep several generations. A perfectly replicated snapshot taken after accidental deletion or malicious mutation faithfully preserves the damage. Periodic restore tests should select more than just the newest file.

Restore into an isolated control plane

Stop API servers and etcd members that could write to the old cluster. Restore the snapshot to a new data directory and member identity according to the intended new topology. etcd restore creates a new logical cluster; peer names, advertised URLs, and initial-cluster settings must agree with the manifests that will start it.

etcdutl snapshot restore /secure-backup/etcd-2026-08-09.db \
  --name=control-1 \
  --data-dir=/var/lib/etcd-restored \
  --initial-cluster=control-1=https://10.0.0.10:2380 \
  --initial-advertise-peer-urls=https://10.0.0.10:2380

Never point a running member at a partially restored directory or mix restored and old members. Follow the Kubernetes distribution’s supported disaster-recovery sequence; managed control planes may not expose etcd at all and must use their provider’s recovery mechanism.

Kubernetes watches cache object revisions. Restoring an older etcd revision can make controllers believe stale cache state is current. Kubernetes and etcd document a revision-bump procedure, commonly paired with marking objects compacted, so informers re-list rather than silently retaining post-snapshot observations. Apply the procedure supported by the installed versions instead of inventing a large write loop.

Prove application-level recovery

After etcd and the API server are healthy, check member health, API readiness, controller-manager and scheduler leadership, node registration, critical add-ons, admissions, webhooks, Secrets decryption, storage attachments, and representative workloads. Compare object counts and selected critical objects against the backup manifest. Confirm controllers reconcile real infrastructure rather than only returning objects from the API.

Run the exercise on an isolated network with production credentials disabled. Record elapsed time and every manual dependency. The result is a recovery plan only when a second operator can repeat it from the artifacts and the cluster passes application-level acceptance tests.

Related:

Sources:

Comments