Understanding Kubernetes Networking: Services, kube-proxy, and CNI Plugins

Kubernetes networking rests on one foundational rule that everything else is built from: every pod gets its own IP address, and every pod can reach every other pod’s IP directly, without NAT — a genuinely flat network model, quite different from Docker’s default bridge-networking-plus-port-mapping approach.

The fundamental model: flat, routable pod IPs

Unlike a single-host Docker bridge network (where containers get private IPs behind NAT and need explicit port publishing to be reached from outside), Kubernetes requires that pod IPs be routable across the entire cluster — a pod on node A can reach a pod on node B by IP directly, with no NAT involved.

kubectl get pods -o wide

NAME        READY   STATUS    IP            NODE
web-abc123  1/1     Running   10.244.1.5    worker-1
web-def456  1/1     Running   10.244.2.8    worker-2

Making this true across real physical or virtual network infrastructure is exactly the job of a CNI (Container Network Interface) plugin.

CNI plugins: who actually wires this up

Kubernetes itself doesn’t implement pod networking — it delegates to a CNI plugin (Calico, Cilium, Flannel, and others), invoked whenever a pod is created or destroyed, responsible for assigning the pod an IP and making it routable according to whatever underlying strategy that plugin uses (VXLAN overlay tunneling, native BGP-advertised routes, or eBPF-based dataplanes, in Cilium’s case).

kubectl get pods -n kube-system -l k8s-app=calico-node

Different CNI plugins make fundamentally different trade-offs — an overlay network (encapsulating pod traffic inside node-to-node tunnels) works on any underlying network but adds encapsulation overhead; a BGP-based approach avoids that overhead but requires the underlying network fabric to cooperate with route advertisement.

Services: stable identity over a shifting set of pods

Pod IPs are ephemeral — a pod that restarts gets a new one. A Service provides a stable virtual IP and DNS name in front of a dynamic, labeled set of pods, so nothing else in the cluster needs to track individual pod IPs directly:

apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 8080

kubectl get endpoints web

The Service’s selector continuously determines its backing Endpoints (or EndpointSlices at scale) — every pod matching app: web gets added automatically, which is the mechanism that makes a Service’s backend set self-updating as pods are created, destroyed, and rescheduled.

kube-proxy: how a Service’s virtual IP actually routes traffic

A Service’s ClusterIP isn’t backed by any real listening process — kube-proxy, running on every node, watches the API server for Service/Endpoint changes and programs the node’s packet-forwarding rules (via iptables or, in more modern configurations, IPVS or eBPF) to rewrite traffic destined for that virtual IP to one of the real backing pod IPs.

iptables -t nat -L KUBE-SERVICES -n | head -20

This is why a Service’s ClusterIP is only reachable from inside the cluster by default — it only exists as a set of packet-rewriting rules on cluster nodes, not a real routable address anywhere else.

Service types: how traffic actually enters from outside

ClusterIP (the default, internal-only), NodePort (exposes the Service on a static port on every node’s own IP), and LoadBalancer (provisions an actual external load balancer via the cloud provider’s integration) form a deliberate escalation of exposure:

spec:
  type: LoadBalancer
  ports:
    - port: 80
      targetPort: 8080

An Ingress resource sits at a layer above plain Services, providing HTTP-aware routing (host/path-based rules, TLS termination) in front of one or more Services, handled by whichever Ingress controller (NGINX, Traefik, and others) is running in the cluster.

Network policies: default-allow until explicitly restricted

By default, every pod can reach every other pod — Kubernetes’ flat network model has no built-in isolation. NetworkPolicy resources add default-deny, allow-list-based restriction, enforced by the CNI plugin (not all plugins support it — this is a common gotcha when choosing one):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-web-to-db
spec:
  podSelector:
    matchLabels:
      app: database
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: web
      ports:
        - port: 5432

Diagnosing connectivity problems

Because there are several independent layers here — CNI-assigned pod routing, Service/Endpoint mapping, kube-proxy’s rewrite rules, and any NetworkPolicy restrictions — a connectivity problem is diagnosed by walking them in order: can the pod reach another pod’s IP directly (a CNI-layer problem if not), does the Service have populated Endpoints (a label-selector mismatch if not), does kube-proxy’s rule set actually exist on the node (iptables -t nat -L or the IPVS equivalent), and finally, is a NetworkPolicy blocking the specific traffic. Each of those is independently inspectable, which is what keeps “the network is broken” from being a dead end rather than a tractable, layer-by-layer investigation.