Kubernetes API Aggregation: APIService, TLS, and Extension Health

Kubernetes API aggregation lets the primary kube-apiserver proxy a registered API path to an extension API server. Projects such as metrics-server use this mechanism to serve APIs beyond the core Kubernetes groups. It is different from a CustomResourceDefinition (CRD): a CRD lets the primary API server serve and store a new resource type, while aggregation delegates an API group to another server that owns its request handling and backend behavior.

That flexibility adds a second API server, a proxy boundary, service discovery, and a certificate trust relationship. A workload for the extension server can be Running while its aggregated API is unavailable because APIService registration, service endpoints, serving TLS, request-header authentication, or delegated authorization is broken. Diagnose each hop independently instead of treating the extension as one Pod health problem.

Decide whether aggregation is the right extension point

Use a CRD when the requirement is a declarative resource type with Kubernetes-managed storage, schema validation, conversion, and ordinary API-server behavior. Use aggregation when a specialized API implementation must serve its own API group or integrate a pre-existing API server behind the Kubernetes API endpoint. An aggregated server that manages Kubernetes resources is commonly paired with controllers, but the API server and those controllers perform separate roles.

The aggregator runs in-process with kube-apiserver. An APIService object claims a group/version path such as /apis/metrics.example.com/v1 and identifies the extension server that handles it. Once registered, matching client requests are authenticated and authorized by the primary API server, then proxied to the extension. That means APIService registration changes the request path for that group; it is not merely a discovery hint.

Keep the extension API server responsive from every API-server replica. Kubernetes documentation requires discovery requests to round-trip through the extension server in five seconds or less. Slow discovery can affect commands and clients that enumerate API groups even if an individual resource request would otherwise succeed. Measure latency from the control-plane network path, not from a developer laptop that reaches the Service by a different route.

Register a group and version

An APIService binds one version of an API group to a backend Service and supplies the CA bundle the primary API server uses to verify that backend’s serving certificate. The following is a schematic registration; replace the group, namespace, Service, certificate bundle, and priorities with values for the actual API server:

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1.metrics.example.com
spec:
  group: metrics.example.com
  version: v1
  groupPriorityMinimum: 100
  versionPriority: 10
  service:
    namespace: metrics-system
    name: extension-api
    port: 443
  caBundle: <base64-encoded-CA-that-signs-the-serving-certificate>

The APIService name follows the version and group, and the Service target must be reachable from the API server. The serving certificate needs names appropriate for the service identity used by the aggregator. Ensure the caBundle is the PEM-encoded CA chain expected by this connection, not a random cluster CA copied from another purpose. If the extension server rotates its serving certificate, rotate the APIService bundle and backend certificate as one controlled change.

Avoid marking the APIService available before its backing server is ready to serve discovery and resource requests. A Service with no ready endpoints, a wrong port, a mismatch between the certificate and Service DNS name, or an invalid caBundle can make the registered API unusable. Read the APIService condition and message before changing certificate files or restarting components.

Understand the two directions of proxy authentication

There are two TLS relationships to distinguish:

The API server authenticates to the extension server. kube-apiserver presents its proxy client certificate. The extension server validates that certificate against the request-header client CA and trusts only the expected proxy identity.
The API server authenticates the extension server. The APIService caBundle lets kube-apiserver validate the extension server’s serving certificate before proxying the request.

These are not interchangeable CA bundles. Kubernetes also uses --client-ca-file for normal client certificate authentication. Reusing the same CA in --client-ca-file and --requestheader-client-ca-file without understanding the matching rules can cause ordinary API clients and control-plane components to be rejected. Keep CA roles separate and document which issuer signs each certificate.

For the first relationship, the kube-apiserver is configured with a proxy client key and certificate, the CA used to validate proxy clients, and allowed certificate common names. The accepted proxy certificate name should be explicit. Kubernetes documents that a blank --requestheader-allowed-names value accepts any common name signed by the configured CA, which is a broader trust rule than most production deployments need.

After authenticating the proxy, the extension server must use the trusted request headers to recover the original caller’s username, groups, and extra attributes. It should not trust arbitrary X-Remote-* headers from a direct unauthenticated connection. Standard extension-server libraries can implement much of this flow, but operators still need to verify how the chosen server reads the extension-apiserver-authentication ConfigMap and applies its configured trust settings.

The extension API server is responsible for authorizing the original caller. A common pattern is delegated authorization: the extension server submits a SubjectAccessReview to the Kubernetes API using a service account allowed to do so by the system:auth-delegator ClusterRole. It also needs permission to read the authentication ConfigMap through the extension-apiserver-authentication-reader role. A successful TLS handshake proves only that the proxy is trusted; it does not prove that the original user is authorized for the requested resource.

Control-plane configuration and managed clusters

The aggregation layer depends on kube-apiserver settings such as --proxy-client-cert-file, --proxy-client-key-file, --requestheader-client-ca-file, --requestheader-allowed-names, and the username, group, and extra-header names. Some distributions configure these automatically. Managed Kubernetes providers may not let tenants set control-plane flags directly, and the provider may own the relevant certificates and authentication ConfigMap.

Before troubleshooting flags, establish who owns the API server. On a self-managed cluster, inspect the actual manifest or process arguments and certificate files using the control-plane administration workflow. On a hosted service, use the provider’s supported API-extension documentation and diagnostics; do not attempt to patch a managed API server’s static manifest or assume a local flag change will persist.

If aggregation is newly enabled on a self-managed control plane, roll out control-plane configuration using the distribution’s supported procedure and maintain access to a known-good control-plane state. Validate one control-plane instance at a time where the architecture permits. A mismatch between proxy certificate, request-header CA, allowed names, and extension server trust can affect aggregated APIs and can also disrupt normal certificate-based API clients when CAs are incorrectly reused.

Troubleshoot from API registration to backend

Start with the APIService condition and test the exact group/version path:

kubectl get apiservice
kubectl describe apiservice v1.metrics.example.com
kubectl get --raw /apis/metrics.example.com/v1

Read the APIService condition’s status, reason, and message. A condition of Available=False is a useful signal that the aggregated API is not ready, but the message should be correlated with control-plane and backend evidence. If discovery fails, check whether the APIService is registered and whether the group and version path match the extension server’s discovery document.

Then walk the network and TLS path:

kubectl -n metrics-system get service extension-api -o yaml
kubectl -n metrics-system get endpointslice \
  -l kubernetes.io/service-name=extension-api -o wide
kubectl -n metrics-system get pods -o wide

Confirm the Service selector, ready EndpointSlices, serving port, network policy, and control-plane route. Compare the endpoint certificate’s SANs and issuing chain with the Service reference and APIService.spec.caBundle. Look at extension-server logs for rejected proxy certificates, untrusted request headers, failed SubjectAccessReview calls, and discovery latency. If the control plane is managed, ask the provider for control-plane-to-Service connectivity evidence rather than testing only from a workload Pod.

Separate common failure classes:

APIService missing or wrong group/version: the path is not registered, or clients are requesting a different version.
APIService unavailable: inspect its condition message, Service endpoints, control-plane routing, and backend readiness.
TLS verification failure: validate the serving certificate SAN, certificate chain, expiry, and APIService CA bundle. Do not solve a trust mismatch by disabling verification.
Proxy authentication failure: confirm the kube-apiserver proxy certificate, its issuer, the extension server’s trusted request-header CA, and the allowed common name.
Identity or authorization failure: confirm the authentication ConfigMap, header names, extension-server delegated-auth configuration, service-account RBAC, and SubjectAccessReview response.
Slow or partial discovery: measure extension server response time from the control plane, verify each API-server replica can reach all endpoints, and inspect backend saturation or dependency latency.

Avoid changing several CA files and APIService fields at once. Capture the current APIService, certificate metadata, API-server configuration source, extension-server logs, and condition messages first. Apply one reversible change, then retest discovery and an authorized resource operation. A successful GET /apis/... is not the same as a successful read or write of the extension’s resources.

Operate upgrades and outages deliberately

Treat the extension API server as a critical API dependency. Monitor APIService availability, extension server readiness, request latency, error rates, certificate expiry, and delegated authorization failures. Test from each control-plane network path during upgrades and after Service, DNS, certificate, or CNI changes. If the APIService becomes unavailable, clients of that extension API can fail even while core API groups remain healthy.

Coordinate versions of the extension server, its API schema, clients, and any associated controllers. Keep old served versions available during a migration window where possible, and test discovery clients that cache group/version information. Do not remove an APIService until clients and controllers have migrated and the objects it serves have a deliberate archival or conversion path.

Use a canary upgrade that checks both /apis/<group>/<version> discovery and a representative authorized resource request. Test an unauthorized identity as well, ensuring delegated authorization rejects it. Observe APIService conditions and extension server logs throughout the rollout. Have a rollback plan for the server image, certificates, and registration object; restoring only the Pod image will not repair an invalid trust bundle.

Production acceptance checklist

Aggregation is chosen for a real custom API-server requirement; a CRD is preferred when API-server-managed storage and schema are sufficient.
The APIService claims the intended group/version and points to a reachable Service with ready endpoints.
Discovery stays within Kubernetes’ documented five-second round-trip requirement from the API server.
The serving certificate, APIService caBundle, proxy-client certificate, request-header CA, and allowed proxy identity are validated as distinct trust roles.
The extension server trusts forwarded identity headers only from the authenticated proxy and delegates authorization for the original caller.
Required ConfigMap and SubjectAccessReview permissions are scoped to the extension server’s service account.
APIService availability, control-plane-to-backend connectivity, delegated auth, and authorized/unauthorized operations are tested during upgrades.
Managed control-plane configuration is changed only through the provider’s supported process.

API aggregation is powerful precisely because it inserts another server into the Kubernetes API path. Reliability depends on treating registration, network routing, two TLS directions, identity forwarding, delegated authorization, discovery latency, and extension-server readiness as separate contracts that must all hold at once.

Related:

Sources: