How the Kubernetes Scheduler Actually Places Workloads

When you kubectl apply a pod, deciding which node it actually runs on is the job of a single, replaceable component: kube-scheduler. Its process is a well-defined two-phase pipeline — filtering out nodes that can’t work, then scoring the ones that remain — and understanding both phases is what turns “why did my pod land there” from a mystery into a traceable decision.

Phase 1: filtering — which nodes even qualify

The scheduler starts with every node in the cluster and eliminates any that can’t satisfy the pod’s hard requirements: insufficient CPU/memory (requests), a nodeSelector or required node affinity that doesn’t match, a toleration mismatch against a node’s taints, or a port conflict with something already running there.

resources:
  requests:
    cpu: "500m"
    memory: "256Mi"

kubectl describe node worker-3 | grep -A5 "Allocated resources"

A pod requesting more CPU than any single node has available (not just installed — already-allocated resources count against this) simply never passes the filtering phase on that node, regardless of how good a fit it would otherwise be.

Taints and tolerations: nodes opting out by default

A taint on a node repels pods unless they carry a matching toleration — the mechanism behind dedicating nodes to specific workloads (GPU nodes, for instance) without needing every other pod in the cluster to explicitly avoid them:

kubectl taint nodes gpu-node-1 dedicated=ml-workloads:NoSchedule

tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "ml-workloads"
    effect: "NoSchedule"

Phase 2: scoring — ranking the nodes that survive filtering

Every node that passes filtering gets scored by a set of scoring plugins, each contributing points based on a specific concern — spreading pods evenly across nodes (NodeResourcesBalancedAllocation), preferring nodes with more free resources left over (NodeResourcesFit), respecting pod affinity/anti-affinity preferences, and more. The node with the highest combined score wins.

kubectl get events --field-selector reason=Scheduled

Affinity and anti-affinity: expressing “near” and “away from”

Pod affinity and anti-affinity let you express placement relative to other running pods rather than absolute node properties — spreading replicas of the same deployment across failure domains (anti-affinity) or co-locating a cache alongside the service that uses it (affinity):

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
            - key: app
              operator: In
              values: ["web"]
        topologyKey: "kubernetes.io/hostname"

topologyKey determines the granularity — spreading across hosts, availability zones, or any other labeled grouping the cluster’s nodes carry.

Priority and preemption: what happens under resource pressure

Pods carry a priority class, and when a high-priority pod can’t be scheduled due to resource pressure, the scheduler can preempt (evict) lower-priority pods elsewhere to make room — a deliberate, auditable trade-off, not a silent one:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000

Custom schedulers: replacing the pipeline entirely

Because kube-scheduler is just a controller watching for unscheduled pods and writing a nodeName back, it’s fully replaceable — a cluster can run multiple schedulers simultaneously, with each pod choosing which one handles it via schedulerName, useful for specialized batch or ML workload schedulers with placement logic the default scheduler doesn’t support:

spec:
  schedulerName: my-custom-scheduler

Diagnosing a pod stuck in Pending

kubectl describe pod on a pod that never gets scheduled shows the scheduler’s own recorded reasoning in its Events section — almost always naming the specific filtering step every node failed (insufficient CPU, a taint with no matching toleration, an affinity rule with no satisfying node), which turns scheduling failures from a black box into a direct, per-node breakdown of exactly why:

kubectl describe pod my-pending-pod
# Events: 0/5 nodes are available: 3 Insufficient cpu, 2 node(s) had taint...

Why this two-phase design matters

Separating hard filtering from soft scoring is what lets Kubernetes express both absolute constraints (“this pod cannot run without a GPU”) and soft preferences (“prefer spreading these pods across zones, but don’t fail if you can’t”) in the same system without conflating the two — a pod either survives filtering everywhere and gets ranked, or it’s genuinely unschedulable, and that distinction is exactly what kubectl describe pod’s event log is designed to surface directly.

Requests, limits, and actual usage

The scheduler primarily reasons from resource requests and other declared constraints, not from a prediction of future application usage. A pod with no meaningful request can be packed onto an already busy node; an exaggerated request can remain Pending while real utilization is low. Establish requests from measured demand and include init containers, sidecars, extended resources, huge pages, and ephemeral storage according to Kubernetes rules.

Limits influence runtime enforcement and quality of service but do not simply replace requests in every scheduling calculation. CPU can be throttled; memory pressure can lead to OOM kills or eviction. Diagnose placement separately from runtime pressure. Metrics Server or monitoring data informs tuning, while the scheduler’s event explains the declared constraint it could not satisfy.

Topology spread and disruption

Topology spread constraints express desired distribution across labeled domains and can be hard or soft. They are often clearer and more scalable than extensive inter-pod anti-affinity for replica spreading. Labels must accurately represent zones or hosts, and maxSkew, unsatisfiable behavior, and eligible domains should be tested during node loss and scale-up.

A successful initial spread does not guarantee continued availability. Node drains, autoscaler decisions, persistent-volume topology, PodDisruptionBudgets, and replacement capacity affect rescheduling. A PDB constrains voluntary eviction; it does not force the scheduler to find a node and does not protect against involuntary failure. Maintain spare capacity or an autoscaling path compatible with hard constraints.

Preemption has a blast radius

PriorityClasses should represent business recovery order, not team status. Preemption removes lower-priority pods to make a node feasible for a pending pod, but nominated placement can change and eviction takes time. Critical workloads still need disruption handling and dependency capacity. Set preemptionPolicy: Never where priority should influence queue order without evicting other work.

Monitor repeated preemption and Pending age; they indicate capacity or policy mismatch. Avoid extremely high default priority, which can turn ordinary applications into universal preemptors. System-critical priority classes are reserved for cluster components and should not be copied into workloads.

A scheduling investigation

Capture the pod specification, scheduler name, recent events, node labels and taints, allocatable and requested resources, volume topology, quota, and admission mutations. Check hard affinity, selectors, tolerations, topology spread, host ports, and extended resources. Then examine autoscaler events: adding a node is impossible when no node group or provisioner can satisfy the constraints.

For custom scheduler profiles, preserve plugin configuration and weights. A profile changes scoring and possibly filtering for every matching pod. Roll it out with representative fixtures and compare placement, availability, and cost. Scheduler logs can help, but the final audit should explain the decision from declared policy rather than depend on one transient log line.

Related:

Sources: