Skip to content
SRE & DevOpsDeep Dive Published Updated 7 min readViews unavailable

Prometheus Staleness Semantics: Lookback, Disappearing Series, and Query-Time Evidence

Understand how Prometheus instant selectors, stale series, lookback, and absence functions affect dashboards and alerts without mistaking missing data for zero.

Prometheus does not treat an absent time series as a numeric zero. That distinction is easy to miss on a dashboard: a line that stops, a query that returns an empty vector, and a line at zero represent different evidence. They can result from an application emitting a real zero, a target disappearing, a label set changing, a query filtering everything out, or a scrape/rule path ceasing to produce a sample. Good diagnosis starts by understanding which of those conditions the query engine can actually observe.

This article is about PromQL query-time behavior and operational interpretation. It is separate from alert-rule state, notification delivery, and promtool rule-fixture authoring. Use those other layers to test the final behavior; do not infer an outage solely from an empty result.

Instant selectors are time-aware, not “last value forever”

At an evaluation timestamp, an instant-vector selector returns the newest sample at or before that timestamp only when it falls inside the lookback period. Prometheus documents a five-minute default, configurable with --query.lookback-delta or an individual query’s lookback_delta parameter. A range query is evaluated repeatedly at its requested steps, so the same selector can return a value at one step and no element at a later step.

For example:

http_requests_total{job="checkout",instance="checkout-2"}

This is not equivalent to asking whether the process is currently reachable. It asks for the selector’s latest eligible sample at each evaluation time. Use a separately understood signal such as up to investigate scrape success, and remember that even up says whether a scrape was successful for a target, not whether every application dependency is healthy.

The lookback window is also not a promise that every old value remains valid for exactly five minutes. When Prometheus has marked a series stale, an evaluation after that marker returns no value for that series; the stale marker takes precedence over merely reusing an earlier sample. The configured lookback matters when selecting ordinary samples, while staleness communicates that the series should no longer participate in the result. Keep these two mechanisms distinct when explaining a gap.

Why a series can disappear

Prometheus documents staleness when a scrape or rule evaluation stops returning a previously present series, and when a target is removed. Once a query evaluation is after the stale marker, that series contributes no value. If samples later arrive again for the same label set, the series can appear again. A visible gap therefore describes a change in query-visible series membership, not a diagnosis of its cause.

Common explanations include:

  • the process stopped exporting that metric or the target became unavailable;
  • the target was removed from service discovery;
  • a scrape or metric relabeling rule dropped the sample;
  • a label changed, producing a different time series rather than continuing the old one;
  • a recording rule stopped producing a result;
  • the query’s matchers, aggregation, or time range no longer select the series.

These cases require different investigations. Compare the raw selector with the expression used by the dashboard, inspect up for the relevant target labels, and review discovery and relabeling state. If an aggregate remains present after one input series disappears, that is expected: aggregation can produce an output from the remaining inputs. Conversely, an aggregate that vanishes can mean every selected input vanished, not necessarily that the application returned zero.

Empty, zero, and stale are different contracts

Suppose a service exports queue_depth. A sample with value 0 says the exporter reported an observed queue depth of zero. An empty instant vector says no series matched or survived query evaluation. It does not mean zero. A stale series is a lifecycle signal that prevents a previously known series from being treated as current. None of these alone establishes why the condition occurred.

This matters for arithmetic and dashboards. A query such as:

sum(queue_depth{service="worker"})

may produce no result if there are no matching elements. Do not silently convert that to zero unless the metric contract justifies the substitution. A presentation layer can intentionally display an empty result as zero, but that is a visualization choice and can conceal instrumentation failure. For alerting, decide explicitly whether “no matching series” should be healthy, unhealthy, or unknown, and test that decision against target removal and label changes.

Use absence functions narrowly

Prometheus provides absent() and absent_over_time() to express missing-vector conditions. absent(v) returns an empty vector if its argument has at least one element; otherwise it returns a one-element vector with value 1. For simple selectors, Prometheus can infer labels for that output from the input selector. absent_over_time(v[10m]) returns a one-element vector with value 1 only when no matching samples occur in the range; if any samples occur, it returns an empty vector.

Examples:

absent(up{job="checkout",instance="checkout-2"})

absent_over_time(
  application_build_info{job="checkout",cluster="west"}[15m]
)

The first asks whether that exact target series is absent at the evaluation time. The second returns 1 only if no matching build-info samples occurred in the selected interval; otherwise it returns an empty vector. Neither tells you whether the exporter crashed, relabeling removed the metric, or a deployment changed labels. A broad selector can also hide a partial failure: if the expression matches any series, absent() is empty even if one expected instance is missing. When the desired contract is “every expected instance exists,” compare observed identities against an authoritative expected inventory rather than relying on one global absence check.

Do not assume a dynamic label value can be reconstructed by absent(). Its label inference is limited to simple matchers documented by Prometheus. For complex expressions, aggregations, or a dynamic set of expected instances, define and test the expected identity model explicitly.

Exporters with their own timestamps are a special case

The Prometheus documentation calls out exporters that attach timestamps to samples. By default, a series that stops being exported can continue to show its last value for the lookback period before disappearing. The track_timestamps_staleness setting changes this behavior. This is not a universal property of every exporter or every scrape: check whether the exporter provides explicit timestamps and the server configuration in effect before explaining a graph gap.

When a chart appears to retain a value after a process has stopped, inspect the scrape configuration, exporter timestamp behavior, and track_timestamps_staleness. Avoid “fixing” the graph by increasing lookback until you know which source of freshness it represents. A longer lookback can make an old value appear current for longer; it cannot repair a missed scrape or establish the health of the underlying component.

A repeatable query investigation

For one missing line, record the exact evaluation timestamp, Prometheus version, query expression, range step, and relevant lookback override. Then proceed from the narrowest evidence outward:

  1. Query the raw metric with the exact labels at an instant where the graph has a gap.
  2. Compare the preceding and following evaluation points, not just the rendered line.
  3. Check up for the matching scrape target and inspect target discovery state.
  4. Compare raw data with each relabeling and recording-rule boundary.
  5. Check whether labels changed in a rollout, creating a new series identity.
  6. If the metric uses explicit timestamps, verify its exporter and staleness configuration.
  7. Decide whether empty data is acceptable, a coverage failure, or an unknown state for the consuming alert or dashboard.

This workflow prevents a common category error: treating the visual absence of a line as proof that the represented resource is down. It also prevents the inverse error of treating a last observed value as current merely because a graph still draws it.

Preserve the semantics in tests and reviews

Prometheus rule tests support _ for a missing sample and stale for a stale sample in test input. Those fixtures let reviewers distinguish “no sample was supplied” from an explicit stale marker while checking the resulting rule expression. The separate rule-testing guide covers fixture mechanics; here, the important review question is whether the expression’s expected result matches the operational contract for an absent series.

When reviewing a dashboard or alert, ask: Is zero meaningful? Which label set is expected? Can a label change replace the series? Is a missing target distinguishable from a missing metric? Is the query instant, range, or an aggregate? Does the expression have an explicit policy for empty output? Record the answer near the query so a future maintainer does not add or vector(0) simply to make a graph look continuous.

Staleness is a query semantic, not a root-cause code. Use it to understand why a series stops participating in an evaluation, then combine it with target, instrumentation, rule, and deployment evidence to explain what happened.

Related:

Sources:

Comments