Kubernetes API Priority and Fairness: Protecting the Control Plane Under Load

Kubernetes API Priority and Fairness, usually shortened to APF, protects the API server when clients collectively ask it to do more work than it can complete safely. It classifies incoming requests, places limited traffic into priority levels, and shares a bounded concurrency budget among those levels. When configured well, an overloaded controller or an aggressive list operation cannot consume every execution slot needed by cluster administration and recovery.

APF is not an authorization system and it is not a billing-oriented rate limiter. A request must still pass authentication, authorization, and admission. APF answers a different question: when many authorized requests arrive at once, which work may execute now, which work waits, and which work receives an HTTP 429 response?

FlowSchemas classify the request

Every request is compared with FlowSchema objects in ascending matchingPrecedence order. The first matching schema wins. Its rules may consider the authenticated subject, resource or non-resource URL, API group, version, resource, namespace, and verb. The schema then names a PriorityLevelConfiguration that controls treatment.

This first-match behavior makes precedence part of the policy. A broad rule placed ahead of a narrow emergency rule can capture traffic before the intended rule is considered. Review the ordered set as one classifier, not as unrelated YAML files.

apiVersion: flowcontrol.apiserver.k8s.io/v1
kind: FlowSchema
metadata:
  name: production-deployments
spec:
  matchingPrecedence: 500
  priorityLevelConfiguration:
    name: workload-writes
  distinguisherMethod:
    type: ByNamespace
  rules:
    - resourceRules:
        - apiGroups: ["apps"]
          apiVersions: ["v1"]
          resources: ["deployments"]
          verbs: ["create", "update", "patch"]
          namespaces: ["production"]
      subjects:
        - kind: Group
          group:
            name: system:serviceaccounts:delivery

A schema’s distinguisher divides matching traffic into flows. ByUser keeps one authenticated user from directly sharing a flow with another. ByNamespace is useful when the same controller works across tenants. The distinguisher is not an identity or security boundary; it is an input to queue selection and fairness.

Priority levels spend concurrency seats

A priority level is either Exempt or Limited. Exempt traffic is not constrained by the normal APF concurrency limit and should remain exceptional. Making routine clients exempt merely moves overload outside the mechanism intended to contain it.

Limited levels receive a share of the API server’s total concurrency according to their nominalConcurrencyShares. Kubernetes measures work in seats rather than assuming every request costs the same. An ordinary write may occupy one seat, while a large list can be estimated to use several. Watch requests have distinct handling because their initial processing and long-lived delivery do not consume the same resources in the same way.

Shares are relative weights, not reserved thread counts and not a promise of latency. Unused capacity can be lent to other levels. Borrowing and lending limits let an operator bound how far a level may move away from its nominal allocation. Start from observed demand and service objectives rather than copying large share values that only change ratios.

Queueing uses shuffle sharding

When a limited priority level reaches its concurrency allowance, it can reject new work or queue it. Rejection is simple and gives the client an immediate 429 response. Queueing absorbs short bursts but consumes memory and introduces latency, so it needs explicit bounds.

APF’s queue configuration includes the number of queues, a handSize, and a per-queue length limit. A flow hashes to a small hand of candidate queues and joins the shortest one. This shuffle-sharding design reduces the chance that a noisy flow shares all of its queues with an unrelated flow. Increasing the number of queues improves isolation at a memory cost; increasing the hand size improves placement choices but increases the probability that two flows overlap.

A queued request is not guaranteed to succeed. It can wait until its turn, time out, or be rejected when its queue is full. Controllers must honor 429 responses and Retry-After instead of retrying in a tight loop. APF can contain a retry storm only if clients cooperate with backoff.

Preserve recovery paths deliberately

The built-in suggested and mandatory objects give core Kubernetes traffic a starting policy. Some objects are maintained by the API server, with an apf.kubernetes.io/autoupdate-spec annotation controlling whether their specifications are reconciled. Editing a built-in object without understanding that behavior can result in a change being overwritten or in losing future recommended defaults.

Create separate objects for workload-specific policy where possible. Before deployment, inventory controllers, CI identities, human administrators, health checks, leader-election calls, and node traffic. Classify read-heavy discovery separately from writes that must remain available during an incident. Keep an authenticated administrative path that has been tested under saturation, but avoid an unlimited catch-all exemption.

APF configuration itself passes through the API server. A policy that starves its own repair identity can make recovery difficult. Store known-good manifests outside the cluster, maintain a tested direct administrative route, and stage changes while load is controlled.

Observe selection, queueing, and rejection

Do not infer APF behavior from object existence. The API server exports flow-control metrics for request execution, seat utilization, queue length, wait duration, rejections, and dispatched work. Break them down by flow schema and priority level, while controlling metric cardinality in long-term monitoring.

The API server can also return X-Kubernetes-PF-FlowSchema-UID and X-Kubernetes-PF-PriorityLevel-UID response headers. They reveal which live objects selected a request and are valuable when a client receives an unexpected delay or 429. Debug endpoints described by Kubernetes can show the current policy and request state, but access to them should be restricted like other control-plane diagnostics.

Load-test with realistic identities and request shapes. A stream of cheap GET calls does not model a controller performing large LIST operations or expensive mutations. Confirm that intended flows land in the correct schemas, a saturated tenant remains isolated, critical reconciliation continues, and clients back off after rejection.

Tune the system, not one YAML field

Start with a hypothesis: which class must retain service, which traffic may wait, and which client should be rejected first? Add alerting for sustained queue delay and rejection, not just CPU. A growing queue may expose trouble before the API server is fully saturated.

Then change one relationship at a time. Raising a priority level’s shares takes relative capacity from others. Enlarging queues can turn visible 429 responses into hidden tail latency. Marking a client exempt can preserve that client while worsening the incident for everyone else. Record the expected effect and a rollback threshold for each adjustment.

A successful APF policy is visible during normal operation and boring during failure. Operators can identify the schema and priority level selected for a request, critical paths keep enough seats to recover the cluster, noisy flows remain bounded, and every rejected client backs off instead of amplifying the overload.

Related:

Sources: