Skip to content
SRE & DevOpsDeep Dive Published Updated 7 min readViews unavailable

Kubernetes CSI Volume Snapshots: Lifecycle, Restore, and Recovery Testing

Operate Kubernetes volume snapshots with CSI prerequisites, binding and deletion semantics, restore workflows, consistency boundaries, and tested recovery.

Kubernetes volume snapshots provide API objects for requesting and representing a point-in-time copy of a persistent volume. They are not a universally available core storage operation, and creating a VolumeSnapshot object does not prove that a usable backup exists. The feature depends on the snapshot CRDs and controller being installed by the distribution, a CSI driver that implements snapshot operations, and the CSI snapshot sidecar that coordinates with the driver. The storage provider may impose additional consistency, retention, and topology constraints.

Treat the snapshot API as one component of a recovery plan. A successful snapshot status means the driver completed the requested storage operation according to its implementation; it does not by itself prove application-level consistency, cross-volume consistency, retention outside the cluster, or that the data can be restored into a running service.

Understand the snapshot resources and prerequisites

VolumeSnapshot is a namespaced request object analogous to a user claim. VolumeSnapshotContent is a cluster-scoped representation of an actual storage-system snapshot, and VolumeSnapshotClass describes driver-specific snapshot parameters and deletion policy. These APIs use the snapshot.storage.k8s.io API group and are installed as CRDs rather than being part of the core Kubernetes API.

Before designing automation, verify that the cluster has the required CRDs, snapshot controller, validating webhook, CSI driver, and csi-snapshotter sidecar. Check the distribution’s installation guidance and the driver’s support matrix. A driver can support CSI volumes without implementing snapshot capabilities. An API object may be accepted by the API server even when the data-plane prerequisites are incomplete, then remain pending or report an error during reconciliation.

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: payments-before-migration
  namespace: production
spec:
  volumeSnapshotClassName: csi-retain
  source:
    persistentVolumeClaimName: payments-data

This requests a dynamically provisioned snapshot of the named PVC. Use a class whose driver matches the CSI driver managing that volume, and do not assume a default class exists. The example is not a substitute for confirming that the selected driver and backend support snapshots or that the claim is in a state where the snapshot operation can be completed.

Track binding and readiness as a lifecycle

The snapshot controller binds the request to a VolumeSnapshotContent; the CSI sidecar asks the driver to create or delete the provider snapshot. Inspect status.boundVolumeSnapshotContentName, status.readyToUse, and any error information exposed in status and events. Do not start a restore merely because the VolumeSnapshot exists. Wait for the readiness condition required by your driver and recovery workflow, and handle timeouts without deleting the only evidence of a failed operation.

For a pre-provisioned snapshot, an administrator creates the VolumeSnapshotContent describing a snapshot that already exists in the storage system and binds it to a VolumeSnapshot. This is a different trust boundary from dynamically requesting a new snapshot. Validate handles, driver identity, namespace/claim references, and whether the object represents the intended source volume before making it available to users.

The deletionPolicy on VolumeSnapshotClass controls what happens to the underlying snapshot when its VolumeSnapshotContent is released. Delete requests deletion through the driver; Retain leaves the provider snapshot and content available for manual reclamation. Neither setting should be chosen casually. Delete can remove the recovery point when Kubernetes objects are cleaned up, while Retain can leave billable data and requires an owner and cleanup process.

Snapshot retention must be considered alongside namespace deletion, PVC deletion, cluster recreation, and provider account lifecycle. A snapshot that is retained by the CSI driver but referenced only by a deleted cluster is not automatically a complete recovery plan. Keep an inventory outside the cluster that maps business recovery points to provider identifiers, source claims, timestamps, driver, retention class, and restore validation results.

Restore into a new claim and validate it

A VolumeSnapshot can be used as a PVC data source to provision a new volume from the snapshot. The destination storage class and topology must be compatible with the driver and the location where snapshot data is accessible. The requested capacity should satisfy the source and driver constraints. A restore can fail even when the snapshot was ready, for example if the destination class is unavailable, the snapshot is not accessible in the selected zone, or the driver cannot create another volume from that source.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: payments-restore-test
  namespace: recovery-test
spec:
  storageClassName: csi-standard
  dataSource:
    name: payments-before-migration
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 80Gi

The example assumes that the snapshot is available in the destination namespace and that the requested storage size and class are supported. A VolumeSnapshot is namespaced, so cross-namespace restore requires the documented data-source reference mechanism and the appropriate reference authorization; do not assume a claim in another namespace can name it directly. Verify the API version and driver behavior on the target cluster before deploying recovery automation.

Restore tests should mount the new claim in an isolated workload, validate filesystem integrity, application-level records, permissions, encryption, and required secrets/configuration, then run representative read and write tests. Do not attach the restored volume to production writers before checking whether the snapshot can be safely modified or whether the original and restored volumes can coexist without identity conflicts.

Distinguish crash consistency from application consistency

A storage snapshot captures storage state at a point in time according to the provider and CSI driver’s behavior. If an application has unflushed buffers, active transactions, multiple data files, or several PVCs that must represent one coordinated instant, a block-level or filesystem-level point-in-time copy may not be application-consistent. Quiesce the application, use database-native backup/checkpoint procedures, or coordinate writers through a documented hook before requesting the snapshot.

Taking independent snapshots of multiple claims does not guarantee they share one atomic timestamp. For a database with data and transaction-log volumes, a mismatch can make recovery impossible or require a more complex replay. Kubernetes documents external CSI volume group snapshot work separately; confirm driver and platform support rather than treating a loop over VolumeSnapshot objects as an atomic group operation.

After a snapshot operation, record the workload version, database checkpoint or LSN where applicable, PVC UID, snapshot UID, CSI driver, and provider snapshot ID. These details help distinguish a technically successful snapshot from a consistent application recovery point. Avoid putting secrets or sensitive payloads in metadata labels or annotation values that are widely readable.

Monitor and recover operationally

Alert on snapshots that remain unready beyond the expected driver window, failed content binding, missing source claims, and repeated CSI controller errors. Include controller, sidecar, and provider event evidence in the runbook. Do not delete a stuck VolumeSnapshotContent finalizer as a generic fix: the finalizer may be protecting a real provider object from orphaning or an underlying delete operation from being skipped.

Test both Retain and Delete behavior in a disposable cluster, including deleting the request object, deleting the source claim, namespace teardown, and restoring after a cluster-level incident. Verify actual provider retention rather than relying on the Kubernetes object’s disappearance. Measure snapshot duration, restore duration, and application validation duration against the recovery time objective.

Finally, schedule restore drills. A snapshot that has never been restored is an untested hypothesis. Restore into an isolated namespace or account, compare known data, exercise application startup, and document every manual dependency. Store independent backups outside the failure domain that contains the cluster and its CSI credentials; snapshots are useful recovery points, but they are not automatically off-cluster backups.

Set a recovery-point objective that accounts for snapshot cadence, quiesce time, and the storage system’s completion latency. A snapshot created every hour cannot recover the last five minutes of writes unless another log or replication mechanism covers that gap. Likewise, a volume restore time is only one part of the recovery-time objective: provisioning, attach, filesystem checks, application replay, secret restoration, DNS cutover, and validation all consume time. Measure the full path during a drill and record the slowest stages rather than extrapolating from a successful API status.

Restrict who can create, retain, restore, and delete snapshots. A principal that can delete both the workload and its recovery points can defeat a cluster-local rollback plan. Use provider-side retention or a separate backup system for recovery from control-plane compromise or accidental deletion, and periodically reconcile the Kubernetes inventory against the storage provider’s inventory so orphaned or missing snapshots are visible.

Related:

Sources:

Comments