Fixing Prometheus Cardinality Explosions Before They Exhaust Memory and Storage

Prometheus stores one time series for each unique metric name and label-set combination. A label such as user_id, full URL, request ID, stack trace, or unbounded container identifier can turn one useful metric into millions of series. The resulting memory growth, write pressure, slow queries, and disk expansion are a data-model failure—not something retention tuning alone can cure.

Confirm cardinality rather than blaming sample volume

Start with Prometheus health, ingestion rate, active series, head memory, WAL and block storage, scrape duration, and rule/query latency. The HTTP API exposes TSDB statistics, including series counts by metric and label pairs, when enabled by the running version:

curl -s 'http://prometheus:9090/api/v1/status/tsdb?limit=20'

Use the expression browser or API to rank metrics deliberately. Avoid running an unbounded aggregation over every series during an already overloaded incident; diagnostic queries themselves consume resources. Preserve configuration, target metadata, Prometheus version, time of onset, and the deployment that changed instrumentation.

Find the dimension that multiplies

Cardinality is multiplicative. A histogram with ten buckets and labels for method, route, status, region, tenant, and pod can create a large cross product even when each label seems harmless alone. Histograms also emit _bucket, _sum, and _count series.

Look for label values that grow with traffic rather than with a bounded system inventory. Route templates such as /users/{id} are bounded; raw paths such as /users/49281 are not. HTTP status code is bounded; an exception message is not. Kubernetes service and namespace labels can be reasonable, while Pod UID or ephemeral container ID may be needlessly expensive for a service-level metric.

Stop the source with the smallest reversible change

The best fix is changing the instrumented application or exporter to remove or normalize the unbounded label. If that cannot ship quickly, use a targeted metric_relabel_configs rule at scrape time to drop the specific metric or label before ingestion:

metric_relabel_configs:
  - source_labels: [__name__]
    regex: 'http_request_duration_seconds'
    action: drop

That example drops the entire metric and is intentionally blunt. A safer real rule should match the exact offending exporter/job and only the problematic family or label. Validate with promtool check config, stage it on one replica when architecture permits, and keep the rollback. Relabeling after ingestion cannot refund memory already used by active head series.

Do not drop job, instance, or other identity labels globally without understanding alert and query semantics. Collapsing distinct series onto the same label set can produce duplicate-sample errors or merge data that should remain separate.

Recovery takes longer than stopping ingestion

Once new offending series stop arriving, existing head series remain until they become stale and age through the head block. Old block data remains until retention removes it or an explicitly managed deletion process rewrites it. Restarting Prometheus may briefly alter memory layout but does not delete the cardinality embedded in WAL and blocks.

Emergency deletion APIs and manual block removal carry data-loss and consistency risks. Preserve evidence and follow the supported storage procedure; never delete arbitrary TSDB directories from a running server. If disk exhaustion is imminent, expand storage or move nonessential data through a tested process while the instrumentation fix propagates.

Redesign the metric contract

Keep metric dimensions finite and useful for aggregation. Put request IDs and detailed error text in logs or traces, then connect signals with exemplars or stable correlation identifiers where supported. Publish a label budget and review new metrics like an API: names, units, allowed values, ownership, expected series count, and removal plan.

expected series ≈ targets × methods × route_templates × status_classes × buckets

This estimate catches explosions before deployment. Add CI checks against a fixture /metrics output, dashboard active-series growth by job, and alert on sudden series churn as well as absolute count.

Close the incident with proof

Verify ingestion, active-series growth, memory, compaction, disk forecast, scrape health, rule evaluation, remote-write queues, and important alerts. Compare equivalent traffic before and after the fix. Cardinality is repaired only when the signal remains operationally useful at a bounded cost—not when the metric disappeared and the dashboard became quiet.

Related:

Sources: