Skip to content
SRE & DevOpsDeep Dive Published Updated 7 min readViews unavailable

Kubernetes Service Traffic Policies: Local Endpoints and Source IP

Choose Kubernetes internal and external Service traffic policies with accurate locality, source-IP, health-check, and endpoint-capacity expectations.

A Kubernetes Service provides a stable virtual endpoint for a changing set of backends. Its traffic policy controls which ready endpoints a node-local data plane may select for certain classes of traffic. The Cluster behavior can use endpoints across the cluster; Local restricts selection to endpoints on the node that receives the traffic. These choices affect source address visibility, failure behavior, capacity distribution, and the role of an external load balancer.

Do not treat Local as a generic performance switch. It changes the eligible endpoint set. If a node receives traffic but has no local ready endpoint, the Service can behave as though no backend is available from that node even when healthy Pods exist elsewhere. A locality policy is safe only when the producer of traffic and the endpoint distribution are designed together.

Separate internal and external traffic

internalTrafficPolicy controls routing for traffic originating inside the cluster and addressed to the Service’s ClusterIP. With Cluster or an unset field, all ready endpoints are eligible. With Local, only endpoints on the originating node are considered. A Pod on a node with no local endpoint can therefore fail to reach the Service through that local-only path.

externalTrafficPolicy applies to traffic that reaches a Service from outside the cluster through supported Service paths such as NodePort or LoadBalancer. With Cluster, cluster-wide endpoints are eligible, and the network path may use source NAT. With Local, the node forwards only to local endpoints and avoids masquerading the external client source address in the Kubernetes service proxy path. Whether the original client address reaches the application also depends on upstream load balancer behavior, proxies, and the cluster’s data plane.

apiVersion: v1
kind: Service
metadata:
  name: edge-api
  namespace: production
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  selector:
    app: edge-api
  ports:
    - name: https
      protocol: TCP
      port: 443
      targetPort: 8443

This example is appropriate only when the external load balancer can direct traffic to nodes with local ready endpoints or correctly health-check and remove nodes that cannot serve the Service. Confirm how your provider implements health checks and source-address preservation. The Kubernetes API expresses policy, but a cloud load balancer, kube-proxy mode, or eBPF service implementation still participates in the actual packet path.

Preserve client addresses with the full network path in mind

Applications often use client IP for rate limiting, audit logs, geo-routing, or access policy. Under externalTrafficPolicy: Cluster, node-level forwarding may source-NAT traffic before it reaches a Pod. Local can preserve the packet source at the service-proxy hop, but it cannot restore an address already replaced by an upstream proxy or load balancer. If TLS terminates at a gateway or reverse proxy, the application may instead need a trusted forwarded-address header with a carefully configured proxy trust list.

Do not trust arbitrary X-Forwarded-For or similar headers from the public internet. The boundary that writes the header must be known, and the application must parse the chain according to the actual trusted proxy topology. Validate this independently of the Service policy. Network source addresses and application-level forwarding headers are different data sources with different spoofing risks.

Test what the backend actually observes by sending requests from a controlled external client and logging the source address at the network and application layers. Include a path through every production load balancer, firewall, ingress, and service mesh component. A local cluster test from a Pod may exercise internalTrafficPolicy but tells you nothing about source IP preservation on an external production path.

Plan endpoint distribution and capacity

With externalTrafficPolicy: Local, each receiving node needs local service capacity or a load balancer that restricts routing to nodes with endpoints. A Deployment spread across zones or nodes may leave some nodes empty, especially after rolling updates, autoscaling, or topology constraints. When one node receives disproportionate traffic, its local Pods can saturate while other Pods remain underused.

Align readiness probes, rollout strategy, topology spread, and external health checks. If a rollout temporarily removes a node’s local ready endpoint, confirm that the load balancer stops sending requests before local capacity disappears. If a cluster uses daemonized node-local proxies or a gateway deployment with one replica per eligible node, document that assumption and alert on endpoint drift.

For internal locality, internalTrafficPolicy: Local is useful only when the workload intentionally co-locates clients and servers and can tolerate a missing local endpoint. It is not a service mesh, a topology scheduler, or a guarantee that local traffic has lower end-to-end latency. Measure real paths and preserve a fallback policy when locality is an optimization rather than a correctness requirement.

The Service API also exposes trafficDistribution for locality preferences that are softer than strict traffic policies. Current Kubernetes documentation describes preferences such as PreferSameZone and PreferSameNode; availability and supported values must be checked against the API server version and distribution. A preference can be useful when the goal is to favor nearby endpoints while retaining broader fallback behavior. It does not promise that every request stays local, and it does not provide the source-address semantics of externalTrafficPolicy: Local.

Keep the decision tied to an observable topology. A zone preference is only meaningful when node labels, endpoint topology, and client distribution are accurate. A node-local policy is stricter: a node with no local ready endpoint cannot use a healthy remote endpoint through that Service path. During scale-out, node maintenance, or a rolling update, endpoint membership can change faster than an upstream load balancer’s target health state. Make the provider’s health-check interval and withdrawal behavior part of the availability budget, and test the interval between a local endpoint becoming unready and the external device stopping traffic to that node.

For a service with a shared external load balancer, decide whether the balancer distributes evenly across nodes or only to nodes with ready local targets. An even per-node distribution can overload a node with fewer local replicas while other nodes have spare capacity. Monitor requests, active connections, errors, and local endpoint count per node; cluster-wide averages can hide a hot or empty node. If local endpoint capacity is not guaranteed by placement and health checks, Cluster routing is often the clearer availability contract even when it changes the source address observed by the Pod.

EndpointSlices remain the control-plane representation of Service backends, while the service proxy or replacement programs the data path. Inspect both layers when debugging. A ready EndpointSlice endpoint does not prove that the receiving node’s proxy has converged or that an external provider has updated its target set. Conversely, an external load balancer can have a healthy node target while the Service has no local endpoint for a particular node.

Diagnose black holes and asymmetric behavior

When a request fails only through a load balancer, check the Service’s policy, node receiving the request, local ready endpoints, EndpointSlice conditions, node health-check state, and provider target configuration. Compare requests to nodes with and without a local endpoint. Confirm the application’s listen address and target port independently; traffic policy cannot repair a process that binds only to loopback or a selector that matches no Pods.

Useful inspection commands include:

kubectl get service edge-api -n production -o yaml
kubectl get endpointslice -n production \
  -l kubernetes.io/service-name=edge-api -o yaml
kubectl get pods -n production -l app=edge-api -o wide
kubectl describe service edge-api -n production

Capture a baseline before changing the policy. Changing from Cluster to Local can immediately reduce the endpoint pool seen by nodes and alter which source address the application receives. Test under realistic node skew, Pod termination, zone loss, autoscaler scale-out, and rollout conditions. Monitor per-node request rate and error rate, not just cluster-wide totals.

Decide with an explicit policy matrix

Choose Cluster when cluster-wide endpoint reachability and even pooling are more important than preserving a direct source address at the node proxy. Choose Local when the network design requires local endpoints, source IP preservation at the service proxy matters, and endpoint distribution plus health checking can guarantee that traffic reaches capable nodes. For internal calls, use internalTrafficPolicy: Local only when co-location is a deliberate architecture property and clients can handle nodes with no local backend.

Document the service type, caller class, proxy path, expected source address, data-plane implementation, health-check owner, and fallback behavior. Revisit that matrix after changing CNI, kube-proxy mode, load balancer, service mesh, or topology. The policy is a routing contract shared by Kubernetes and the surrounding infrastructure, not an isolated YAML preference.

Related:

Sources:

Comments