Skip to content
SRE & DevOpsDeep Dive Published Updated 6 min readViews unavailable

Understanding Kubernetes Networking: Services, kube-proxy, and CNI Plugins

How pod-to-pod networking, Services, and kube-proxy's packet rewriting fit together to make Kubernetes' flat network model actually work.

Kubernetes networking rests on one foundational rule that everything else is built from: every pod gets its own IP address, and every pod can reach every other pod’s IP directly, without NAT — a genuinely flat network model, quite different from Docker’s default bridge-networking-plus-port-mapping approach.

The fundamental model: flat, routable pod IPs

Unlike a single-host Docker bridge network (where containers get private IPs behind NAT and need explicit port publishing to be reached from outside), Kubernetes requires that pod IPs be routable across the entire cluster — a pod on node A can reach a pod on node B by IP directly, with no NAT involved.

kubectl get pods -o wide
NAME        READY   STATUS    IP            NODE
web-abc123  1/1     Running   10.244.1.5    worker-1
web-def456  1/1     Running   10.244.2.8    worker-2

Making this true across real physical or virtual network infrastructure is exactly the job of a CNI (Container Network Interface) plugin.

CNI plugins: who actually wires this up

Kubernetes itself doesn’t implement pod networking — it delegates to a CNI plugin (Calico, Cilium, Flannel, and others), invoked whenever a pod is created or destroyed, responsible for assigning the pod an IP and making it routable according to whatever underlying strategy that plugin uses (VXLAN overlay tunneling, native BGP-advertised routes, or eBPF-based dataplanes, in Cilium’s case).

kubectl get pods -n kube-system -l k8s-app=calico-node

Different CNI plugins make fundamentally different trade-offs — an overlay network (encapsulating pod traffic inside node-to-node tunnels) works on any underlying network but adds encapsulation overhead; a BGP-based approach avoids that overhead but requires the underlying network fabric to cooperate with route advertisement.

Services: stable identity over a shifting set of pods

Pod IPs are ephemeral — a pod that restarts gets a new one. A Service provides a stable virtual IP and DNS name in front of a dynamic, labeled set of pods, so nothing else in the cluster needs to track individual pod IPs directly:

apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 8080
kubectl get endpoints web

The Service’s selector continuously determines its backing Endpoints (or EndpointSlices at scale) — every pod matching app: web gets added automatically, which is the mechanism that makes a Service’s backend set self-updating as pods are created, destroyed, and rescheduled.

kube-proxy: how a Service’s virtual IP actually routes traffic

A Service’s ClusterIP isn’t backed by any real listening process — kube-proxy, when used, watches the API server for Service and EndpointSlice changes and programs the node’s supported packet-forwarding backend. Some CNI dataplanes replace kube-proxy with their own eBPF implementation; that is a separate architecture, not an eBPF mode inside kube-proxy.

iptables -t nat -L KUBE-SERVICES -n | head -20

This is why a Service’s ClusterIP is only reachable from inside the cluster by default — it only exists as a set of packet-rewriting rules on cluster nodes, not a real routable address anywhere else.

Service types: how traffic actually enters from outside

ClusterIP (the default, internal-only), NodePort (exposes the Service on a static port on every node’s own IP), and LoadBalancer (provisions an actual external load balancer via the cloud provider’s integration) form a deliberate escalation of exposure:

spec:
  type: LoadBalancer
  ports:
    - port: 80
      targetPort: 8080

An Ingress resource sits at a layer above plain Services, providing HTTP-aware routing (host/path-based rules, TLS termination) in front of one or more Services, handled by whichever Ingress controller (NGINX, Traefik, and others) is running in the cluster.

Network policies: default-allow until explicitly restricted

By default, every pod can reach every other pod — Kubernetes’ flat network model has no built-in isolation. NetworkPolicy resources add default-deny, allow-list-based restriction, enforced by the CNI plugin (not all plugins support it — this is a common gotcha when choosing one):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-web-to-db
spec:
  podSelector:
    matchLabels:
      app: database
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: web
      ports:
        - port: 5432

Diagnosing connectivity problems

Because there are several independent layers here — CNI-assigned pod routing, Service/Endpoint mapping, kube-proxy’s rewrite rules, and any NetworkPolicy restrictions — a connectivity problem is diagnosed by walking them in order: can the pod reach another pod’s IP directly (a CNI-layer problem if not), does the Service have populated Endpoints (a label-selector mismatch if not), does kube-proxy’s rule set actually exist on the node (iptables -t nat -L or the IPVS equivalent), and finally, is a NetworkPolicy blocking the specific traffic. Each of those is independently inspectable, which is what keeps “the network is broken” from being a dead end rather than a tractable, layer-by-layer investigation.

EndpointSlices and traffic readiness

EndpointSlices are the scalable API for Service backends. They carry addresses, ports, readiness and topology information used by proxies and controllers. A Service with the right selector can still have no ready endpoints because pods are not Ready, labels differ, ports are wrong, or the controller has not produced slices. Inspect the Service, EndpointSlices, pod readiness, and application listener together.

Readiness gates traffic but does not authorize it. internalTrafficPolicy, externalTrafficPolicy, session affinity, topology-aware routing, and source-IP behavior change routing semantics and must be tested against the current Kubernetes and cloud-provider implementation. Preserve these settings in incident evidence; two Services with the same selector can behave differently.

DNS and ingress are separate layers

CoreDNS normally resolves Service names from Kubernetes data. A DNS failure can originate in pod resolver configuration, CoreDNS capacity, upstream resolvers, network policy, or node networking. Test the fully qualified Service name, inspect resolver configuration, and measure latency and error rate rather than treating every name failure as a Service failure.

Ingress and Gateway API require controllers; creating an API object without a matching controller does not provision a data plane. The controller may create cloud load balancers, health checks, addresses, and firewall rules. Record which implementation owns each resource, where TLS terminates, and how status conditions report acceptance. A LoadBalancer Service or Gateway can also consume provider quota and take time to converge.

Policy and observability

NetworkPolicy is additive: once a pod is isolated for a direction, allowed peers and ports come from the union of applicable policies. Namespace and pod selectors have precise semantics, and policy enforcement depends on the networking implementation. Start with explicit default-deny policies in a test environment, add DNS and required platform flows, and verify with positive and negative connection tests.

Collect CNI and kube-proxy or replacement-dataplane health, dropped-packet evidence, EndpointSlice changes, DNS metrics, conntrack pressure, load-balancer health, and application traces. A packet capture may be necessary, but capture at the correct pod, node, overlay, or load-balancer boundary and protect payload data. Define an MTU test because overlay encapsulation can produce failures that small health probes do not reveal.

Related:

Sources:

Comments