Prometheus Native Histograms: Sparse Buckets, Mergeable Resolution, and Migration

Classic Prometheus histograms turn every configured bucket boundary into a separate time series, plus sum and count. They are easy to inspect and broadly supported, but bucket choices made in application code determine future precision and each additional label combination multiplies the bucket series.

Native histograms add a structured histogram sample to the Prometheus data model. One sample contains count, sum, a zero bucket, and sparse positive and negative buckets governed by a schema. Prometheus introduced the feature experimentally in 2.40 and documents it as stable beginning with 3.8, while ingestion and remote write still require explicit configuration in current 3.x releases.

Sparse buckets pay for populated ranges

A standard native histogram uses exponentially spaced bucket boundaries. Instead of storing every possible bucket, it encodes spans and counts for populated regions. Empty ranges can therefore cost very little, while the schema controls resolution across a large numeric range.

The sample also has a zero bucket for values around zero, separate positive and negative sides, count, sum, and optional exemplars. This makes native histograms suitable for more than strictly positive latency, although instrument semantics must still be chosen carefully.

Sparse does not mean free. A metric with a very broad or adversarial distribution can create many populated buckets, and each sample is larger than one float. Set instrumentation and scrape-side bucket limits based on measurements.

Resolution can be reduced during aggregation

Standard schemas are designed so a higher-resolution histogram can be converted to a lower compatible resolution. When PromQL aggregates native histograms with different schemas, the result can use a common lower resolution instead of requiring every producer to share an exact handwritten boundary list.

That is a major operational advantage, but the least detailed input can determine useful output resolution. Storing one service at extreme precision brings little value if every dashboard immediately aggregates it with lower-resolution peers.

Choose a bucket factor from the smallest distinction users need, not from a desire to retain every measurement. Add limits such as native_histogram_bucket_limit and a minimum bucket factor at ingestion so one target cannot impose unbounded cost.

PromQL treats the histogram as a sample

For a native counter histogram, apply rate() to the histogram first, then aggregate and derive a statistic. A percentile query no longer needs the classic le label:

histogram_quantile(
  0.95,
  sum by (service) (
    rate(http_request_duration_seconds[5m])
  )
)

Functions such as histogram_count(), histogram_sum(), histogram_avg(), and histogram_fraction() extract estimates or totals from native histogram values. Counter-reset handling belongs inside rate() on the full histogram, so calculating a sum first and then rating it can produce incorrect behavior.

PromQL can annotate incompatible or suspicious operations. Inspect query warnings during migration rather than checking only whether a graph rendered a line.

Counter and gauge histograms are distinct flavors

Most request-duration histograms are counter-like: bucket populations, sum, and count accumulate until reset. Native histograms also model gauge-like distributions whose bucket values can move arbitrarily between samples, such as a snapshot of item ages.

Use rate() or increase() for counter histograms and delta() for gauge histograms according to the documented semantics. A function may technically accept an unsuitable flavor and emit an annotation, but that does not make the result meaningful.

Instrumentation libraries should expose flavor correctly. Renaming a metric does not convert a snapshot distribution into an accumulating counter.

Scraping requires protocol negotiation

In current Prometheus 3.x configuration, scrape_native_histograms: true enables ingestion and changes negotiation to prefer a protobuf exposition format that carries native histogram data. A target and client library must actually expose that data; the server option cannot invent it from arbitrary classic buckets by default.

Remote write has a separate send_native_histograms: true setting. Verify the receiver supports the protobuf fields before enabling it. A receiver that ignores unknown fields can silently drop the only representation if classic series were already removed.

Record the negotiated scrape protocol and confirm stored sample types with a query. Configuration accepted by promtool is necessary but not proof that the complete path preserves histograms.

Dual publication makes migration observable

Many client libraries can expose classic and native representations during transition. Prometheus can be configured to ingest both when the endpoint provides both. This temporarily costs more, but it creates a comparison window for dashboards, recording rules, alerts, remote storage, federation, and API consumers.

Run equivalent queries side by side over the longest range used in production. Percentiles are estimates in both models and need not be numerically identical because boundaries and interpolation differ. Define an acceptable error based on the service-level objective, especially near an SLO threshold.

Do not switch a 30-day dashboard after collecting native samples for only one day. The Prometheus specification recommends enough parallel history to cover the query windows before replacing classic queries.

Cardinality changes shape rather than disappearing

A classic histogram creates one series per bucket and label set. A native histogram usually creates one time series per label set, with a richer sample. That can sharply reduce series count, but storage, memory, and network results depend on distribution, scrape interval, resolution, and compression.

High-cardinality labels remain high cardinality. Putting user IDs or request URLs into a native histogram still creates a distinct structured series for each value. Native buckets solve the bucket-series dimension, not uncontrolled labels.

Benchmark head memory, WAL volume, block size, query CPU, remote-write bytes, and receiver ingestion. Keep classic-vs-native comparisons tied to the same traffic period and retention.

Build a reversible rollout

Start with one bounded metric and one Prometheus shard. Enable client publication, scrape ingestion, optional dual classic collection, then remote write only after receiver validation. Add recording rules and dashboards under new names so rollback does not require editing every consumer at once.

Test counter resets, mixed schemas, empty and zero-heavy distributions, negative observations where allowed, a bucket-limit reduction, missing native samples, and a target falling back to text exposition. Use promtool for rule tests and inspect API warnings.

Remove classic publication only after every retention window and downstream integration has crossed the migration. Keep a documented switch-back until that older data ages out.

Native histograms make distributions first-class Prometheus values and allow resolution to compose during aggregation. Their payoff is not automatic. It comes from bounded instrumentation, correct flavor-aware PromQL, verified protocol support, and a parallel migration long enough to prove that no alert or remote system lost its data.

Related:

Sources: