Skip to content
SRE & DevOpsDeep Dive Published Updated 4 min readViews unavailable

Kubernetes Kubelet Pods API: Local Inventory, Watches, and Failure Boundaries

Use the Kubernetes Kubelet Pods API for node-local Pod state while respecting its socket permissions, startup readiness, rate limits, and watch gaps.

Node agents often need to know which Pods kubelet is actually managing on their own node. Querying the Kubernetes API server for that local view adds a control-plane dependency and may return information that has not yet caught up with kubelet’s runtime reconciliation. The PodsAPI feature gate is Alpha and disabled by default in Kubernetes v1.36, then Beta and enabled by default in v1.37. In v1.36, the API is available only when that gate is explicitly enabled; do not infer its presence from the version alone.

Treat the release boundary as part of the design: v1.37 is the first release where this API is Beta and enabled by default, not the first release where it exists. In v1.36 it is Alpha and disabled by default. This node-local interface is not a replacement for cluster-wide API queries, the Kubernetes API server’s authorization model, or the separate PodResources API.

A node-local read interface, not a second cluster API

On supported nodes, the service listens on /var/lib/kubelet/pods-api/pods-api.sock. The socket’s file permissions restrict access to local administrators, typically root. The API is not supported on Windows nodes. Its gRPC service offers three operations:

  • ListPods returns Pods currently managed by kubelet on that node.
  • GetPod retrieves one Pod by UID.
  • WatchPods streams updates when a locally managed Pod changes state.

Requests can include a protobuf FieldMask, such as status.phase or status.podIPs, so a monitor that needs only a small status view need not request full Pod objects. With no mask, the API returns a full v1.Pod. That can expose more workload configuration than a health checker needs, so request only the fields required and keep socket access privileged.

The similarly named PodResources API is a different contract. It reports allocated container resources such as device and CPU information; it is not this Pod-spec-and-status service. Avoid wiring a client for one endpoint into a consumer expecting the other.

Local freshness has a specific meaning

Kubelet serves the most up-to-date Pod information it knows locally, using its cache and reconciliation with the container runtime. This API remains available when a node loses control-plane connectivity. That makes it useful for node-local agents that must keep operating during a network or API-server interruption.

It does not make the node’s view a cluster-wide source of truth. It cannot list Pods on other nodes, authorize users through Kubernetes RBAC, or update Pod objects. A locally current status can also differ temporarily from what the API server has observed. Use the API for local decisions and diagnostics; use the control-plane API for cluster-level inventory and policy.

Handle startup and stream failure as normal states

Immediately after kubelet starts or restarts, its Pod sources may not have completed their initial synchronization. During that interval every operation returns gRPC FAILED_PRECONDITION rather than incomplete data. A consumer should expose an explicit “local inventory not ready” state and retry with bounded backoff; it should not turn an empty or failed response into “there are no Pods.”

ListPods and GetPod are rate-limited by default to 100 queries per second with a burst of 10. Excess requests return RESOURCE_EXHAUSTED. WatchPods is not subject to that rate limit, but a consumer that cannot keep up can have its stream dropped. Kubelet exposes kubelet_pod_watch_events_dropped_total for watch events dropped because a client was too slow.

A resilient node agent should therefore prefer a watch for ongoing changes over tight polling, record RPC status codes, and treat a dropped stream as a reconciliation boundary. Reconnect and compare a fresh local listing with the agent’s own state; do not assume a broken stream replays every event. Keep any retry loop bounded and jittered so many agents do not stampede kubelet after a restart.

For a Linux node, an operator can first verify the endpoint and access policy without changing it:

sudo stat -c '%A %U:%G %n' /var/lib/kubelet/pods-api/pods-api.sock

Then inspect the consumer’s gRPC errors and kubelet’s request/watch metrics before increasing polling or changing socket permissions. A permission failure is a privilege-boundary issue, not evidence that the Pod inventory is empty.

Rollout checklist

  • Confirm the node’s kubelet version is v1.37 or later and the feature is available in that distribution.
  • Grant socket access only to the local administrative component that needs it; the API does not provide Kubernetes RBAC authorization.
  • Use a field mask and stable Pod UIDs rather than names alone when correlating lifecycle events.
  • Handle FAILED_PRECONDITION, RESOURCE_EXHAUSTED, and stream disconnects as distinct observable states.
  • Keep cluster-wide policy and inventory decisions on the Kubernetes API server path.

Related:

Sources:

Comments