Skip to content
SRE & DevOpsDeep Dive Published Updated 8 min readViews unavailable

Kubernetes CoreDNS: Corefile, API Synchronization, and Cache Behavior

Operate Kubernetes CoreDNS by tracing Corefile plugins, API watch synchronization, cache behavior, upstream forwarding, and actionable metrics.

CoreDNS is usually the Kubernetes cluster DNS implementation, but its visible Service keeps the historical name kube-dns for compatibility. CoreDNS resolves Kubernetes service records from data it watches through the Kubernetes API and forwards other names to configured upstream resolvers. The Corefile controls zones and plugins; the Deployment, Service, EndpointSlices, RBAC, and network path determine whether that configuration can actually answer a Pod’s queries.

When a Pod reports DNS failures, distinguish three questions: did the query reach the cluster DNS Service, did the Kubernetes plugin have synchronized and authorized API data, and did an upstream resolver answer the forwarded query? A single CoreDNS Pod in Running state does not establish all three. Diagnose the request path and the specific DNS response before changing a global Corefile.

Trace the Kubernetes DNS path

For a normal cluster record, a Pod sends DNS to the nameserver in its own /etc/resolv.conf. That address commonly represents the cluster DNS Service. The Service routes the query to CoreDNS Pod endpoints, and the Kubernetes plugin answers names from synchronized Service and EndpointSlice state. For a name outside the zones handled by that plugin, a forward plugin can send the query to an upstream resolver.

CoreDNS commonly runs as a Deployment in kube-system; the Service is named kube-dns for compatibility. Confirm both objects and their endpoint data:

kubectl -n kube-system get deployment coredns
kubectl -n kube-system get service kube-dns -o wide
kubectl -n kube-system get endpointslice \
  -l kubernetes.io/service-name=kube-dns -o wide
kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide

If the Service has no ready endpoints, fix its selector or the CoreDNS Pods before changing DNS records. If only one node or replica fails, compare the target Pod’s node, the Service data plane, and the endpoint list at the same time. A successful query from one Pod does not prove that all nodes route to the same DNS endpoints.

The Kubernetes plugin synchronizes watches from the API server. By default it waits up to five seconds for synchronization at startup. If it cannot sync within that interval, it can begin serving while the watches continue to retry; queries for Kubernetes records that are not yet synchronized receive SERVFAIL. The plugin reports readiness after it has synchronized with the API. A readiness endpoint can therefore reveal an initialization or API-access problem that a process-level health endpoint alone will not.

The CoreDNS ServiceAccount also needs the list/watch permissions required by the Kubernetes plugin. Inspect the ClusterRole and binding installed for the exact CoreDNS release, including access to the relevant Services and EndpointSlices. Missing or insufficient API permissions can look like a DNS outage even when CoreDNS accepts queries. Avoid granting broad permissions as a quick fix; compare the installed role with the current distribution or upstream manifest and restore only the needed reads.

Read the Corefile as routing and policy

The Corefile is often stored in a ConfigMap named coredns. A common shape defines the cluster zone, reverse lookup zones, health and readiness endpoints, an upstream forwarder, a cache, and loop detection:

.:53 {
    errors
    health :8080
    ready :8181
    kubernetes cluster.local in-addr.arpa ip6.arpa {
        fallthrough in-addr.arpa ip6.arpa
    }
    prometheus :9153
    forward . /etc/resolv.conf
    cache 30
    loop
    reload
}

This is an explanatory example, not a replacement for a distribution-managed Corefile. The cluster domain may not be cluster.local; preserve distribution-specific options and plugin versions. The kubernetes plugin is authoritative for the configured cluster zones, while fallthrough lets selected unmatched reverse lookups continue through the plugin chain. The forward directive handles names not answered by the Kubernetes plugin in this server block. A stub zone can be routed to a specific internal resolver instead of sending every external name to the same upstream.

health and ready answer different questions. The health endpoint reports that the CoreDNS process is responsive. The ready endpoint waits for readiness-capable plugins; the Kubernetes plugin reports ready after API synchronization. A Deployment probe using only process health may therefore allow a Pod into the Service before it has current cluster records. Check the shipped probes before changing them, and ensure the readiness contract matches the plugins and server blocks in use.

The forward . /etc/resolv.conf target deserves special attention. If the file points at a local stub resolver that forwards back to the cluster DNS address, requests can loop. Inspect the resolver file visible inside CoreDNS and the node’s resolver configuration. Kubernetes documents a systemd-resolved failure mode in which a stub resolv.conf can create a forwarding loop; kubeadm detects that configuration, but other distributions may require their own supported kubelet resolver setting.

Interpret cache behavior before changing TTLs

The CoreDNS cache plugin can cache successful and denial responses. A numeric setting such as cache 30 caps the maximum cache TTL at 30 seconds, but the plugin also applies a default minimum cache duration of 5 seconds to both success and denial responses. An upstream TTL below five seconds can therefore be raised to that minimum; set the MINTTL value in the respective success or denial directive when a different floor is required. Entries can still be evicted sooner when a cache shard reaches capacity. Treat negative-cache duration separately when diagnosing a newly created name that remains unavailable through a resolver.

Negative caching matters during rollouts. If a client asked for a Service name before the record existed, a cached denial can remain visible for the applicable cache duration even after the Service is created. Compare the answer from the CoreDNS Service with the answer from the client process, and consider caches in the language runtime, node-local DNS, or another recursive resolver. Repeatedly lowering every TTL or restarting all CoreDNS Pods can hide which layer held the stale answer.

Stale-serving behavior is a deliberate availability trade-off, not a general DNS repair. Serving an expired record can help when an upstream is unavailable, but it can also preserve an obsolete answer after an endpoint or external address changes. For a cluster Service, assess whether that stale endpoint is safe for the workload before enabling stale cache behavior. Do not use keepttl casually for responses CoreDNS is not authoritative for: the CoreDNS cache documentation warns that downstream resolvers may then retain stale answers.

Monitor cache requests, hits, evictions, and stale answers when the Prometheus plugin is enabled. Interpret the counters by server block and zone where available. A high hit rate can reduce upstream traffic, but does not by itself prove clients are receiving current answers. A rise in stale-served entries, cache evictions, or request latency should be correlated with query volume, record TTLs, upstream health, and application symptoms.

Separate cluster records from forwarded records

Test a known Kubernetes name and an external name through the same resolver path. If a fully qualified Service name works but external names fail, inspect the forward target, upstream reachability, egress rules, and forward-plugin health checks. If external names work but Services fail, inspect Kubernetes plugin readiness, API watch permissions, Service selectors, EndpointSlices, and the configured cluster domain.

Use a temporary diagnostic Pod or an approved debug container with dig or nslookup; do not add tools to a production application image solely for an incident. Record the exact query name, record type, response code, resolver address, source Pod and node, and timestamp. NXDOMAIN usually points to a name or authoritative negative answer; SERVFAIL indicates that a resolver could not complete processing; a timeout points to a missing response along the network path. These are clues, not proofs, so compare them with CoreDNS logs and upstream health.

CoreDNS’s forward plugin maintains upstream health checks after exchange errors and exposes per-upstream request duration and health-check metrics. If all upstreams are unhealthy, the plugin’s default behavior is not identical to an explicit SERVFAIL policy; verify the configured options before inferring an outage from one response. Bound concurrent forwarded queries only after measuring expected query rate, upstream latency, and memory headroom. The plugin documentation notes that requests above a configured max_concurrent limit are rejected with REFUSED, which is materially different from a timeout.

Change configuration with a reversible test

CoreDNS configuration is often propagated through a ConfigMap and reloaded by the reload plugin, but a successful ConfigMap write does not prove every replica has loaded the new configuration. After a change, inspect Pod status and logs for the reload result, then issue a controlled query through the Service and confirm the expected answer and response code. Follow the cluster distribution’s supported configuration workflow rather than directly editing a managed add-on that will be overwritten by reconciliation.

For short-lived query tracing, the log plugin can confirm whether queries reach CoreDNS and show the returned response. It can produce substantial volume and expose internal query names, so enable it only for a bounded diagnostic window, protect the resulting logs, and remove it after testing. Use metrics for ongoing trend analysis and logs for time-bounded evidence rather than leaving every DNS query in a high-volume production log indefinitely.

Before deploying a Corefile change broadly, test the configured zones, external forwarding, reverse lookups, search-expanded names, and negative answers from representative namespaces. Confirm behavior with the actual CoreDNS image version and distribution-provided manifest. Roll out gradually where supported, watch readiness and upstream health, and retain the previous configuration for a prompt rollback. A reliable DNS change preserves the resolver contract for both in-cluster and external names instead of merely making one failing query succeed.

Related:

Sources:

Comments