Kubernetes Lease API: Heartbeats, Leader Election, and Failure Boundaries

Distributed components often need a small, frequently updated record that says who is alive or which replica currently leads. Storing that signal in a full application object creates unnecessary contention and mixes coordination with business state. Kubernetes provides the coordination.k8s.io/v1 Lease object for this narrow role.

The control plane uses Leases for node heartbeats, control-plane components use them for leader election, and newer Kubernetes versions can publish API server identity through them. Workloads may also create their own Leases. The object is a coordination hint with a time window, not a distributed lock that physically prevents a stale replica from acting.

LeaseSpec records a claim and its timing

Important fields include holderIdentity, leaseDurationSeconds, acquireTime, renewTime, and leaseTransitions. A leader-election client writes an identity, renews before the duration expires, and increments transition information when leadership changes.

These fields are not a server-enforced mutex. The API server stores and versions the object; clients interpret time and ownership rules. A buggy or partitioned old leader can continue touching an external database after another client legitimately acquires the Lease.

Use a unique holder identity that distinguishes pods across restarts, such as a generated value containing pod identity, not a shared Deployment name. Keep it free of secrets because users with read access can inspect the object.

Updates rely on optimistic concurrency

A client reads the current Lease, computes whether it can renew or acquire, then updates using the object’s resourceVersion. If another contender writes first, the API server rejects the stale update and the client retries from fresh state.

Do not implement leader election with an unconditional patch that overwrites another holder. Use the maintained client-go leader-election package unless there is a compelling protocol requirement. It already handles acquire, renew, retry timing, conflicts, and callbacks that have accumulated years of production scrutiny.

API request latency belongs in timing design. Set lease duration, renew deadline, and retry period so transient delays can be absorbed while true failure is detected within the product’s objective. Copying aggressive values from a local cluster into a cross-region control plane can cause repeated leadership churn.

Node heartbeats use a dedicated namespace

Each node has a corresponding Lease in the kube-node-lease namespace. Kubelets update these small objects more frequently than the larger Node status, reducing write load while giving the control plane a current liveness signal.

A current Lease says the kubelet could renew through the API path. It does not prove every workload, disk, CNI path, or application endpoint on the node is healthy. Node conditions and controller logic combine heartbeat information with other status before scheduling and eviction decisions.

Monitor missing or delayed renewals, but do not page on one isolated API timeout. Correlate with node readiness, API server latency, network partitions, and controller actions.

Leader-election callbacks define the safety boundary

The client-go elector invokes callbacks when leadership starts, when it stops, and when a new leader is observed. The work launched by OnStartedLeading must use the provided cancellation context. OnStoppedLeading should be treated as loss of authority, not a suggestion to finish an unlimited backlog.

Stop issuing new side effects before advertising a clean shutdown. Bound operations so they can be canceled or made harmless when their result returns late. A goroutine that ignores cancellation can outlive its Lease and overlap with the new leader.

Leadership callback code should also be idempotent. Process restart, API conflict, and network delay can produce transitions at inconvenient points. Persist business progress separately from the Lease.

A Lease does not provide fencing

Fencing means an old leader is technically unable to mutate the protected resource after authority transfers. A Kubernetes Lease alone cannot impose that rule on an external database, cloud API, device, or message broker.

For destructive or non-idempotent work, add a fencing mechanism the resource validates. Options include a monotonically increasing epoch stored with every write, conditional updates against a resource version, a database advisory-lock protocol with connection loss semantics, or work items whose unique IDs make duplicate completion harmless.

Do not use wall-clock comparison in the workload as the only fence. Node clocks can differ, and process pauses can be longer than expected. Lease timing tells clients when to compete; the protected system must reject stale authority where split-brain consequences matter.

RBAC should isolate each coordination domain

A participant needs permission to get, create where appropriate, update, and possibly watch a specific Lease or tightly scoped set. Granting write access to every Lease in a namespace allows one workload to disrupt unrelated elections.

Use a dedicated namespace or exact resourceNames policy when the creation workflow permits it. Remember that create permission cannot be constrained by resourceNames in the same way as updates, so pre-creating Leases can simplify least-privilege design.

Audit identity changes and unexpected transitions. Do not log every healthy renewal at high volume, but expose counters for acquisition attempts, renewal failures, leadership duration, API conflicts, and callback shutdown time.

API availability is part of workload availability

A healthy leader that cannot renew through the API server must eventually step down, even if it can still reach the system it controls. That conservative behavior prevents two replicas from deliberately claiming leadership based on private connectivity views, but it couples operation to Kubernetes API availability.

Choose whether the controller should fail closed, continue a read-only service, or preserve a bounded amount of local work during that event. Document the tradeoff. Increasing the lease duration reduces sensitivity to API interruptions but extends failover and potential stale-leader overlap.

Do not put large payloads or rapidly changing application state in the Lease. That increases conflicts and hides state behind an object intended for lightweight coordination.

Test pauses and partitions, not only pod deletion

Deleting the leader pod is the easy case. Test API server latency, one-way partitions, a process paused beyond the lease duration, CPU starvation, lost watch events, repeated update conflicts, clock skew, graceful termination, and callbacks that refuse to stop.

Verify one invariant at the protected resource: during every forced transition, either only the current epoch can commit or duplicate work is safe. Also measure how long the system has no leader and how long old work survives after leadership loss.

Kubernetes Leases provide an efficient common record for heartbeats and elections. Their reliability depends on optimistic concurrency, realistic timing, cancellation-aware work, narrow RBAC, and an external fencing story. The object can identify a winner; only the rest of the system can make that winner exclusive.

Related:

Sources: