Skip to content
SRE & DevOpsDeep Dive Published Updated 5 min readViews unavailable

Kubernetes CSI Inline Ephemeral Volumes: Node Lifecycle and Attribute Boundaries

Operate Kubernetes CSI inline ephemeral volumes safely by tracing node-side lifecycle calls, scheduling limits, driver attributes, and cleanup failures.

Kubernetes has two CSI-related volume sources that can appear inline in a Pod, but they do not share a lifecycle. A CSI ephemeral volume uses the Pod’s csi volume source and delegates node-local creation and publication to a driver that explicitly supports the mode. A generic ephemeral volume embeds a PVC template; Kubernetes creates a PVC and then follows the ordinary provisioning path. The distinction determines which component creates storage, what the driver must implement, and which failure signals an operator should inspect.

This guide focuses on the CSI inline path, not generic ephemeral PVC provisioning, CSI attachment failures, or snapshots. CSI ephemeral volumes became stable in Kubernetes v1.25, but only a subset of CSI drivers support them. Confirm the cluster release and the specific driver’s support before designing a workload around this feature.

The node service owns the short lifecycle

The Pod is scheduled first. During volume setup on the selected node, kubelet asks the CSI node service to publish the inline volume. In this flow the driver receives NodePublishVolume and later NodeUnpublishVolume; the Kubernetes CSI guidance says the driver creates the ephemeral resource while handling publication and removes it as part of unpublishing. The volume is therefore local to the node selected for the Pod. It is not a remotely provisioned durable volume merely because the CSI driver also supports persistent volumes.

The request context identifies the mode with csi.storage.k8s.io/ephemeral: "true". Inline parameters are passed through from the Pod’s volumeAttributes; they are driver-specific, not a standardized Kubernetes storage schema. Drivers need to treat publication as a lifecycle operation, not just a mount call: if setup partially allocates something and then fails, Kubernetes does not guarantee another NodePublishVolume retry after every failure. A driver must undo partial work before returning an error or implement its own garbage collection. On teardown, it may finish deletion before returning from NodeUnpublishVolume or complete deletion afterward, according to the CSI guidance.

Here is a deliberately illustrative Pod fragment. Replace the example driver name and attributes with those documented by an installed driver; this manifest is not deployable by itself.

apiVersion: v1
kind: Pod
metadata:
  name: image-worker
spec:
  restartPolicy: Never
  containers:
    - name: worker
      image: busybox:1.36
      command: ["sh", "-c", "echo ready; sleep 3600"]
      volumeMounts:
        - name: work
          mountPath: /work
  volumes:
    - name: work
      csi:
        driver: inline.storage.kubernetes.io
        volumeAttributes:
          class: scratch

The Pod API tells Kubernetes which driver and attributes to pass. It does not define what class: scratch means, how large the result is, whether its data is encrypted, or how the driver removes it. Those guarantees belong to the driver’s documented contract and implementation.

Scheduling and accounting are intentionally limited

CSI inline ephemeral storage is prepared after scheduling, when Kubernetes has already selected a node. Kubernetes does not use this source for storage-capacity-aware Pod scheduling. If the chosen node cannot create the volume, startup can remain blocked while kubelet retries setup; the scheduler does not automatically pick a different node because the volume was too large or unavailable. The volume is also outside kubelet’s Pod ephemeral-storage accounting and enforcement, which applies to storage kubelet manages itself.

That makes CSI inline volumes a fit for specialized, lightweight, node-local needs, not a shortcut for ordinary capacity-managed persistent storage. If storage capacity, PVC status, snapshots, cloning, resizing, or scheduler-aware provisioning are requirements, compare the generic ephemeral PVC path or a normal PVC instead. Do not infer resource limits from the container’s ephemeral-storage request or limit and assume they constrain a driver’s separate local allocation.

Treat Pod-supplied attributes as user input

The inline API places volumeAttributes in a Pod specification, so a user who can create that Pod can provide those values to the driver. Kubernetes warns that a driver is unsuitable for inline use if it exposes settings that should be administrator-controlled, such as parameters normally restricted to a StorageClass. The driver should validate every attribute and avoid turning a user-controlled value into an unrestricted host path, privileged device operation, or persistent allocation.

Cluster administrators can disallow inline use for a driver by removing Ephemeral from its CSIDriver.spec.volumeLifecycleModes, or can apply an admission webhook that restricts the driver or selected attributes. Check the actual object rather than inferring support from a chart name:

kubectl get csidriver inline.storage.kubernetes.io -o yaml
kubectl get pod image-worker -n workload -o wide
kubectl describe pod image-worker -n workload
kubectl get events -n workload --sort-by=.metadata.creationTimestamp

The first command shows whether the driver advertises the inline lifecycle mode. The remaining commands help correlate a Pod’s assigned node, volume setup conditions, and namespace events. Event wording and retention vary; compare timestamps and the Pod’s UID when a name has been reused. Then inspect the CSI node-plugin logs for that same node using the labels and container names defined by the driver’s deployment, rather than assuming a vendor-neutral label selector.

A production review checklist

Before rollout, verify all of the following against the exact driver version and cluster:

  • The CSIDriver object advertises Ephemeral, and the node plugin is present on every eligible node.
  • Attributes exposed in a Pod are safe for the Pod author to control; admission policy blocks disallowed drivers or values.
  • The driver has bounded allocation, failure cleanup, idempotent unpublish behavior, and a recovery path for orphaned node-local state.
  • Workloads tolerate node loss and Pod recreation. Ephemeral data follows this volume lifecycle; it is not a backup or durable state store.
  • Capacity, quotas, and monitoring exist outside Kubernetes’ ordinary Pod ephemeral-storage accounting when the driver consumes meaningful node resources.
  • Pod placement constraints describe hardware or topology needs that the scheduler can actually see. They do not manufacture capacity awareness for the inline volume itself.

For an incident, correlate Pod scheduling, kubelet volume setup, the driver’s node-local operation, and cleanup. A Pod stuck after assignment may indicate a driver-side create or publish failure, not a scheduler defect. A successfully removed Pod with leaked node state points instead to unpublish or driver garbage collection. Keeping those lifecycle stages separate makes the next diagnostic action much less speculative.

Related:

Sources:

Comments