Kubernetes NodeLocal DNSCache: Rollout, Routing, and Failure Analysis

NodeLocal DNSCache runs a DNS caching agent on each Kubernetes node as a DaemonSet. Instead of sending every Pod query through the cluster DNS Service and its normal Service-routing path, a Pod can query the local agent. The cache forwards misses to the cluster DNS Service for cluster records or to the configured upstream resolver for external names.

The feature can reduce DNS latency and UDP connection-tracking pressure, but it adds a node-level networking component to the name-resolution path. A partially rolled-out cache, an address collision, a kubelet resolver mismatch, or an OOM-killed cache process can produce node-specific DNS failures that look like application or CoreDNS incidents. Treat deployment as a coordinated change to the node resolver path, not just “install one more DaemonSet.”

NodeLocal DNSCache has been stable upstream since Kubernetes v1.18. Managed distributions may package, configure, or operate it differently, so first establish whether the cluster already has a provider-managed implementation. Do not install a second cache over a platform-managed DNS add-on.

Trace the query path before changing it

With the usual ClusterFirst Pod DNS policy, the kubelet places cluster DNS information in the Pod’s resolver configuration. Without NodeLocal DNSCache, a typical query goes to the CoreDNS Service IP, then through the cluster’s Service dataplane to a CoreDNS endpoint. With NodeLocal enabled, the local caching agent listens on the node and answers cache hits itself. For a cluster-domain cache miss, it forwards to the CoreDNS Service; for other names, it forwards to configured upstream resolvers.

The local hop can avoid the ordinary DNAT and connection-tracking path for Pod-to-DNS-Service requests. The agent can also use TCP when forwarding to CoreDNS while continuing to accept client DNS over UDP, which can reduce the effects of dropped upstream UDP queries without requiring application changes. This is a performance and resilience improvement, not a replacement for CoreDNS: the cluster DNS service still supplies cluster records, and external lookups still depend on the upstream path.

The cluster domain is cluster.local by default, but it can be configured differently. Verify the actual domain and the Service IP in the cluster rather than copying example values. The historical kube-dns Service name may front CoreDNS; its name does not prove that the legacy kube-dns implementation is running.

Kube-proxy mode changes the listener contract

The upstream manifest and kubelet resolver settings differ by dataplane mode:

With kube-proxy in iptables mode, the NodeLocal agent listens on the node-local address and the CoreDNS Service IP. The existing kubelet --cluster-dns setting can normally remain unchanged.
With kube-proxy in IPVS mode, the agent listens only on the node-local address because the CoreDNS Service IP is already used by the IPVS interface. The kubelet’s --cluster-dns setting must then point to the node-local address.

That distinction is an easy source of a cluster-wide outage: the DaemonSet can appear healthy while Pods still send queries to an address where no local listener exists. Identify the actual kube-proxy mode on every relevant node pool, inspect the active kubelet configuration, and verify which IP appears in a test Pod’s /etc/resolv.conf after rollout.

Choose an address that cannot collide with a Pod CIDR, Service CIDR, node address, host route, VPN, or other node-local service. Kubernetes documentation recommends an address with local scope, such as an unused address in 169.254.0.0/16 for IPv4 or an appropriate unique-local IPv6 address. These ranges are examples, not an allocation plan; coordinate with the CNI, host networking, and infrastructure teams. For IPv6, bracket addresses where the manifest uses an IP:port form.

The upstream sample runs NodeLocal DNSCache in hostNetwork mode and manages it as a DaemonSet. Consequently, changes to the sample manifest are not automatically safe for every CNI, distribution, or control plane. Use the provider-supported add-on or validate a customized manifest against the exact node image and kube-proxy implementation.

Plan cache memory and health behavior

NodeLocal DNSCache stores entries and consumes memory for concurrent queries. Its Pods do not watch every Kubernetes Service or EndpointSlice, so cache memory is driven by query patterns and concurrency rather than directly by the number of cluster objects. The current Kubernetes documentation cites CoreDNS’s default cache as 10,000 entries and about 30 MB when full for each server block; actual peak use depends on configured cache blocks, concurrency, and workload behavior. Measure the deployed agent under representative load before setting a restrictive memory limit.

Memory pressure has a DNS-specific failure mode. If the node-local cache container is OOM-killed, its custom packet-filtering rules might remain while the process is down. A DaemonSet restarts the Pod, but queries redirected to an unhealthy local listener can fail during recovery. Alert on cache restarts and OOM events, per-node DNS error rates, and missing metrics; a healthy DaemonSet desired count alone does not establish that every node is resolving names.

Use the CoreDNS max_concurrent forward-plugin setting only when its behavior and memory trade-off are understood for the installed configuration. Reducing cache size or concurrency can lower memory pressure but may increase upstream query load or latency. Tune from observed query volume, cache hit behavior, and peak concurrency, then test both success and degraded states.

Configure cluster and external domains deliberately

The node-local Corefile distinguishes cluster names, reverse lookups, and external names. A cluster-domain miss is forwarded to the CoreDNS Service. External names go to the upstream resolvers discovered or configured for the node-local agent. If a cluster uses stub domains, conditional forwarders, split-horizon DNS, or private corporate zones, verify that those rules are represented in the supported CoreDNS/NodeLocal configuration path.

Some providers do not allow direct edits to the node-local-dns ConfigMap. Kubernetes documents compatibility paths involving the traditional kube-dns ConfigMap format, but provider behavior differs. Avoid manual changes that an add-on manager will overwrite. Store the intended configuration in the owning system, review the rendered Corefile, and verify reload behavior after changes.

Caching also changes how quickly DNS answers reflect upstream changes. DNS TTLs and negative caching determine the practical behavior; the cache is not an application-level connection pool and cannot repair an application that holds stale addresses indefinitely. When changing a Service endpoint, external record, or stub domain, test both positive and negative answers from a workload using its real DNS policy. Consider the effect of TTL, cache contents, and client-side resolver caching before declaring a propagation delay a Kubernetes networking fault.

NetworkPolicy and host-network semantics require care. NodeLocal DNSCache uses host networking, and policy enforcement for host-network traffic is implementation-specific. Do not assume that an ordinary Pod egress rule automatically controls the node-local agent’s upstream queries, or that a default-deny Pod policy blocks its listener in the expected way. Test the actual CNI behavior and separately control the node-level egress path to CoreDNS and upstream resolvers.

Roll out in controlled stages

Before deployment, record the existing kube-dns/CoreDNS Service IP, cluster domain, resolver settings per node pool, kube-proxy mode, CNI behavior, stub domains, and upstream resolver path. Confirm ownership of the DNS add-on. Select a collision-free local address and decide whether the kubelet configuration must change for IPVS.

Deploy first to a representative non-production node pool. Verify the DaemonSet reaches every intended node, including tainted or specialized pools, and that all Pods are Ready. Confirm the address is bound, the expected firewall rules are installed, and the kubelet is sending queries to the right listener. Test same-namespace and cross-namespace Services, fully qualified cluster names, headless Services, reverse lookups if used, stub domains, public names, negative answers, and IPv4/IPv6 behavior where applicable.

Then roll out by node pool or availability zone with a rollback path. Watch DNS latency, timeout rate, CoreDNS QPS, node-local cache metrics, conntrack pressure, DaemonSet restarts, and application request errors. Avoid changing the kubelet resolver target and deleting the old DNS path simultaneously unless the rollout mechanism guarantees a safe transition. In IPVS mode, make the kubelet setting and the local listener change as one coordinated change.

Use commands like these to inspect coverage and compare behavior from an affected workload:

kubectl -n kube-system get daemonset node-local-dns
kubectl -n kube-system get pods -l k8s-app=node-local-dns -o wide
kubectl -n kube-system logs -l k8s-app=node-local-dns --all-containers=true --since=10m
kubectl -n kube-system get service kube-dns -o wide
kubectl exec -n production deploy/api -- cat /etc/resolv.conf

The labels and deployment name can differ in managed clusters; discover the actual add-on objects before using selectors. Query metrics from the node-local DNS Pods and CoreDNS independently so a cache hit is not mistaken for a healthy upstream. Compare a cache hit, a deliberate miss, an internal Service name, and an external name. Run tests on multiple nodes, because node-specific listener or routing problems can be masked when every test Pod lands on the same machine.

Rollback without creating a resolver gap

If the cache must be removed, restore any kubelet DNS settings that were changed, then remove the DaemonSet using the supported add-on mechanism. Verify each affected Pod and node now targets the original CoreDNS Service path. Do not leave kubelets pointed at a local address after the local listener disappears. Conversely, do not delete a provider-managed DaemonSet just because a second manifest failed; establish which controller owns it and revert the configuration through that controller.

After rollback, re-test internal and external queries from each affected node pool, confirm CoreDNS endpoint health and application recovery, and check that stale custom rules or host routes are not intercepting traffic. Preserve the failing node, logs, and metrics long enough to identify whether the fault was listener binding, kubelet configuration, address collision, Corefile forwarding, resource limits, or the upstream resolver.

Production acceptance checklist

Exactly one supported NodeLocal DNSCache implementation owns each intended node.
The local address is unique and routable as designed across IPv4/IPv6 and every CNI path.
Iptables and IPVS behavior are handled explicitly, including kubelet --cluster-dns where required.
Cluster, stub, reverse, and external query paths are tested from real Pods on multiple nodes.
Memory requests/limits and concurrency settings are based on measured peak behavior.
Per-node metrics, DNS errors, OOMs, restarts, and missing DaemonSet coverage alert before users report failures.
Rollback restores the kubelet resolver target and removes the cache without leaving a DNS black hole.

NodeLocal DNSCache is most useful when high query volume, latency, or UDP conntrack behavior justifies another node-level component. Its value comes from shortening the path and making cache behavior observable; its operational cost is a new listener, cache, address, and failure mode on every participating node. Deploy it only when both sides of that trade-off are measured and the resolver path can be verified end to end.

Related:

Sources: