Skip to content
SRE & DevOpsDeep Dive Published Updated 6 min readViews unavailable

StatefulSet PVC Retention: Separate Scale-Down from Deletion

Configure StatefulSet claim retention deliberately, inspect controller ownership, and test scale-down without confusing PVC deletion with volume recovery.

A StatefulSet gives each ordinal a stable identity and, when it uses volumeClaimTemplates, a corresponding PersistentVolumeClaim (PVC). The lifetime of that claim is a separate decision from the lifetime of its Pod. This distinction matters when scaling down a database, deleting a StatefulSet during a migration, or cleaning up a test environment: a disappearing Pod does not necessarily mean its data has been removed, and a disappearing PVC does not necessarily mean its underlying volume is recoverable.

Kubernetes defaults both StatefulSet PVC retention decisions to Retain. The optional .spec.persistentVolumeClaimRetentionPolicy makes the two decisions explicit: whenScaled applies to excess ordinals removed by scale-down; whenDeleted applies when the StatefulSet itself is deleted. Each accepts Retain or Delete. The policy concerns PVCs made from that StatefulSet’s volumeClaimTemplates, not every claim mounted by the Pods.

Read the two axes independently

For a three-replica StatefulSet, reducing replicas to two condemns the highest ordinal. With whenScaled: Retain, its template-created PVC remains for possible reuse if that ordinal returns. With whenScaled: Delete, the controller arranges for that claim to be garbage-collected after its Pod terminates. Deleting the StatefulSet invokes whenDeleted, not a retrospective scale-down of every ordinal. A replacement Pod after node failure is neither a scale-down nor StatefulSet deletion; Kubernetes retains its existing claim.

The controller uses owner references to implement this lifecycle. For whenDeleted: Delete, it sets the StatefulSet as owner of applicable PVCs. For whenScaled: Delete, it first attaches an owner reference to each condemned Pod’s template-created PVC, then removes the Pod. Garbage collection follows termination so that unmounting precedes claim deletion. This is an eventual controller workflow, not a synchronous storage erase at the instant an operator changes a replica count.

Inspect before changing policy

Use read-only checks in the target namespace:

kubectl get sts ledger -n app -o yaml
kubectl get pods,pvc -n app -o wide
kubectl get pvc data-ledger-2 -n app -o yaml
kubectl get pv -o wide

Confirm the StatefulSet’s current replicas, claim template names, PVC owner references, bound PV, StorageClass, and PV reclaim policy. The last setting determines what happens to the backing PV after a PVC is deleted; it is not overridden by the StatefulSet retention setting. Snapshot, backup, and application-consistency requirements likewise live outside this field.

A manifest can state the intent explicitly:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: ledger
spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Retain

This fragment is deliberately incomplete; add it to a valid StatefulSet manifest rather than applying it alone. Check that the cluster version supports the field. The feature is stable and enabled by default as of Kubernetes 1.32; older clusters may require the StatefulSetAutoDeletePVC feature gate on both API server and controller manager. Provider support and upgrade state should be verified against the actual cluster.

Treat automatic deletion as a data-loss policy

Delete is useful for disposable per-ordinal data, but it is a poor substitute for a retention schedule. Before adopting it for persistent workloads, prove restoration from a backup on a separate environment, document which ordinals can be discarded, and check the StorageClass/PV reclaim behavior. Test a controlled scale-down and observe Pod termination, PVC owner-reference changes, PVC disappearance, and PV disposition. Do not test on irreplaceable production data.

There is also a controller-failure edge case. Kubernetes warns that force-deleting a condemned Pod while the StatefulSet controller is down may race the owner-reference update. Wait for reconciliation when possible; if intervention is unavoidable, inspect the affected PVCs’ owner references before assuming the configured deletion policy executed. A leftover claim is not proof of a safe backup, and a missing claim is not proof that the storage backend has securely erased its data.

The safe operational model is three layered decisions: StatefulSet policy governs template-created PVCs, PVC deletion interacts with PV reclaim policy, and the storage backend and backup system govern recoverability. Record each layer before changing replica count or deleting the controller.

Diagnose a claim that appears stuck

After a planned test, an old claim may remain Bound, enter Terminating, or vanish while the PV remains. These are different states. First compare the desired replica count with the actual Pod list and the claim name. A scale-down from three to two targets ordinal 2, not ordinal 0. If ordinal 2 still has a Pod, the controller may be waiting for ordered termination or an application shutdown. The claim should not be expected to disappear before the Pod is gone.

If the Pod is gone but the PVC persists under a Delete policy, inspect the PVC’s metadata.ownerReferences and the StatefulSet controller’s health. Do not manually strip owner references merely to make the symptom disappear; they are the controller’s deletion contract. If the PVC is Terminating, inspect metadata.finalizers and check for another Pod still using it. Kubernetes’ PVC protection postpones removal while a Pod object refers to the claim. This is a safety mechanism, not necessarily a failed garbage collector.

If the PVC disappears but the PV remains, compare the PV’s persistentVolumeReclaimPolicy and status. Retain leaves the PV in Released for manual reclamation; it does not automatically make old data available to a new claimant. With a supported Delete reclaim policy, Kubernetes and the provisioner should remove the PV and backend asset. A PV finalizer may keep the API object visible while deletion of the storage asset is being completed. Investigate provisioner logs and provider state before assuming a timeout means the asset is safe to reuse or delete by hand.

For a repeatable exercise, record kubectl get sts,pods,pvc -n app -o wide before and after a scale-down in a disposable namespace. Also record the bound PV name from kubectl get pvc data-ledger-2 -n app -o jsonpath='{.spec.volumeName}'. Observe transitions rather than relying on one final snapshot. Restore the replica count only after deciding whether the old claim should be reused or a new one provisioned. If the policy deleted the claim, scaling back up is not a data restore; it may create an empty volume under the same ordinal name.

Plan migration and rollback explicitly

Changing retention policy does not back up or copy data. Before a migration, record the StatefulSet manifest, every template-created claim/PV pair, the storage provisioner, reclaim policy, backup identifiers, and application recovery procedure. Prefer a tested restore into a separate namespace or cluster as evidence of recoverability. If a workload needs its data after controller deletion, keep whenDeleted: Retain and verify the claims remain before rebuilding the controller. For a deliberately disposable test workload, Delete can automate cleanup, but document which storage asset deletion it may trigger.

The identity of an ordinal is stable, but its data is stable only while the corresponding claim and backing volume are retained and usable. This is why a scale-down policy should be reviewed alongside the scaling mechanism: a manual kubectl scale, an autoscaler, and a GitOps manifest applying an older replica count can all change the desired ordinal set. The StatefulSet documentation notes that applying a manifest with .spec.replicas can overwrite a prior manual scale. Treat configuration drift as a potential data-lifecycle event when whenScaled: Delete is enabled.

Related:

Sources:

Comments