Prometheus Alert Rules: Evaluation, Pending State, and Notification Boundaries

A Prometheus alert is the result of a periodically evaluated query, not a continuous event subscription and not proof that an operator was notified. The lifecycle has distinct boundaries: a rule group schedules PromQL evaluations, Prometheus tracks the state for each returned label set, and Alertmanager groups and routes alert updates to notification receivers. Treating those as one step makes timing, missing-data, and delivery failures difficult to diagnose.

Groups define the evaluation clock

Prometheus loads rule files through rule_files. A group can set an interval; otherwise it inherits the global evaluation_interval, which defaults to one minute. Rules in the same group share an evaluation time and run sequentially by default. This ordering matters when a recording rule feeds a later rule in that group. Put rules that must observe the same recorded values together, and isolate expensive or unrelated work when its evaluation budget should not delay other rules.

query_offset moves a group’s query evaluation timestamp into the past. This gives recently scraped or remotely written samples time to arrive before a rule reads them; it does not add samples or repair a late pipeline. Choose an offset from measured ingestion delay and account for the resulting freshness lag. If a group has not finished before its next interval, Prometheus skips that iteration until the prior run completes or times out. Watch the exported prometheus_rule_group_iterations_missed_total counter (the rule documentation names it without the prometheus_ namespace prefix), group evaluation duration, and rule errors; a syntactically valid rule that repeatedly misses its schedule is not operating as intended.

Groups execute concurrently by default. Within one group, the rules remain sequential unless the --enable-feature=concurrent-rule-eval feature is enabled; then Prometheus may run rules it determines to be independent in parallel. This is an opt-in feature, not an automatic capacity fix: more parallel queries can raise CPU, memory, and query contention. Verify support and the applicable --rules.max-concurrent-evals setting against the exact server release before enabling it, then measure the whole rule workload.

`for` tracks one alert identity across evaluations

An alert expression returns an instant vector. Every returned element, identified by its labels, is an active alert instance. With for: 10m, a new instance starts Pending and becomes Firing only after Prometheus continues to see that same identity through the required duration. The timer is not a moving-window query; use a PromQL range such as [5m] when the condition itself needs historical samples. Firing is recognized at an evaluation, so detection is quantized by the group schedule and can take longer than the configured for duration in wall-clock time.

Labels are part of identity. If a templated or expression-derived label changes between evaluations, Prometheus sees a different alert instance, so the new instance starts its own pending period. Keep labels bounded and stable, such as service, cluster, team, and severity; put changing values and explanatory text in annotations instead. Annotations can use $labels and $value to add instance context and the evaluated value without creating new alert identities.

groups:
  - name: checkout-slo
    interval: 30s
    query_offset: 30s
    rules:
      - alert: CheckoutHigh5xxRatio
        expr: |
          (
            sum by (service) (rate(http_requests_total{service="checkout", code=~"5.."}[5m]))
            /
            sum by (service) (rate(http_requests_total{service="checkout"}[5m]))
          ) > 0.02
          and on (service)
          sum by (service) (rate(http_requests_total{service="checkout"}[5m])) > 1
        for: 10m
        keep_firing_for: 2m
        labels:
          severity: page
          team: commerce
        annotations:
          summary: "Checkout 5xx ratio is above 2%"
          description: "Checkout exceeds the error-ratio threshold at {{ $value }}."
          runbook_url: "https://runbooks.example/checkout/high-5xx"

This example assumes http_requests_total is a counter with service and three-digit HTTP code labels. The traffic guard suppresses a low-volume ratio, while the 5xx filter and denominator use the same metric family. Replace the example runbook address and tune the threshold from the service’s SLO, not by copying it unchanged.

Clearing, keeping, and missing data

When an expression stops returning an alert’s label set, a Pending instance is cleared; a Firing instance normally resolves at that evaluation. keep_firing_for delays resolution for the configured duration after a firing condition disappears, which can reduce flapping. It also intentionally keeps a recovered alert firing longer, so it is not a substitute for correct queries or a missing-data policy. It starts only after an alert has fired, and it should be tested against both short recovery blips and genuine recovery.

No result is not the same as a healthy zero. A stale series is excluded from later queries; otherwise, instant selectors use the newest sample within the lookback period, five minutes by default. A scrape failure, removed target, renamed label, or never-created metric can therefore make an alert expression return no elements. Exporters that attach their own sample timestamps are an important exception: after a series stops being exported, its last value can remain queryable for the default five-minute lookback before disappearing; track_timestamps_staleness changes this behavior. For example, absent(up{job="checkout"}) can detect that an entire expected job has no up series; it does not identify each expected service if the job has many members. Use target health (up == 0) and an explicit inventory or absence rule appropriate to the expected population. Alert separately on telemetry gaps instead of relying on keep_firing_for to guess whether an absent signal means recovery.

Prometheus firing is not notification delivery

Prometheus exposes loaded rules and active alert instances in its Rules API and UI. That confirms evaluation state, not successful paging. The /rules API endpoint is useful for interactive checks, but it has weaker stability guarantees than the overarching API v1; automation should tolerate that distinction. Prometheus sends alert updates to configured Alertmanager instances; Alertmanager handles grouping, routing, inhibition, silences, and receiver notifications. A correct expression can still produce no page if the rule is not loaded, Alertmanager is unreachable, labels miss a route, a silence applies, or the receiver fails. Test the rule state and the notification path as separate systems.

Make the lifecycle measurable

Before rollout, run promtool check rules and add promtool test rules cases for the first Pending evaluation, the Firing transition after for, an identity-label change, a brief recovery within keep_firing_for, final resolution, and absent input. Use the promtool version matching the deployed server. Inspect /api/v1/rules?type=alert to confirm the rule is loaded, healthy, and exposing the expected labels and state. During a representative load test, require group evaluation duration to remain below its interval and the exported prometheus_rule_group_iterations_missed_total not to increase; confirm the metric name on the deployed server’s /metrics endpoint. Finally, test both a normal route and a deliberately silenced or unmatched test alert; verify the expected receiver, suppression, and resolution behavior.

keep_firing_for was added in Prometheus 2.42.0; older servers will not accept this rule field. Rule-group settings and feature flags are release-sensitive too. Pin the Prometheus image and its promtool together, validate the rule against that pair, and check the deployed binary’s help and feature flags before using concurrent evaluation.

Related:

Sources:

Groups define the evaluation clock

for tracks one alert identity across evaluations

Clearing, keeping, and missing data

Prometheus firing is not notification delivery

Make the lifecycle measurable

Comments

`for` tracks one alert identity across evaluations