Skip to content
SRE & DevOpsHow-To Published Updated 6 min readViews unavailable

Nomad Canary Deployments: Allocation Gates, Promotion, and Rollback Boundaries

Stage Nomad service-job changes with canary allocations, verify health, and promote deliberately without confusing scheduler health with traffic analysis.

Nomad’s canary setting stages new allocations before the scheduler proceeds with the rest of a destructive update. It is an allocation and deployment gate, not a built-in HTTP traffic percentage. That difference determines what an operator must measure, how much spare capacity the job needs, and what promotion actually means.

This walkthrough is scoped to service jobs and the behavior documented by HashiCorp’s current Nomad job-specification and deployment guides. The command and implementation details below are checked against an immutable Nomad source snapshot at commit bf24b0dd97430d3b364b68b1f7375e85e585778f; that snapshot is evidence for that revision, not a claim that every Nomad release behaves identically. Check the documentation and CLI help for the server/client versions you actually run.

What the canary setting stages

An update block controls a task group’s rolling-update strategy. If an update would destructively replace allocations, canary asks Nomad to create the specified number of new allocations without first stopping the old ones. The deployment remains pending promotion while an operator checks the candidate. Once promoted, the remaining destructive updates proceed at max_parallel.

For a service group with four allocations, canary = 1 means one new allocation is staged first. The old allocations remain in place until promotion, so plan for temporary resource demand above the steady-state count. A canary that cannot be placed because of CPU, memory, port, volume, or constraint pressure is not a successful test. It is a capacity or scheduling problem that must be understood before a rollout proceeds.

job "billing-api" {
  datacenters = ["dc1"]
  type        = "service"

  group "api" {
    count = 4

    network {
      port "http" {
        to = 8080
      }
    }

    update {
      canary            = 1
      max_parallel      = 1
      min_healthy_time  = "60s"
      healthy_deadline  = "10m"
      progress_deadline = "20m"
      auto_promote      = false
      auto_revert       = true
    }

    task "server" {
      driver = "docker"

      config {
        image = "registry.example.invalid/billing-api@sha256:REPLACE_WITH_REVIEWED_DIGEST"
        ports = ["http"]
      }

      service {
        provider = "nomad"
        name     = "billing-api"
        port     = "http"

        check {
          type     = "http"
          path     = "/readyz"
          interval = "10s"
          timeout  = "2s"
        }
      }
    }
  }
}

The image value is deliberately a placeholder using the reserved .invalid domain. Replace it with a reviewed, immutable image digest and validate the job in a non-production environment before using it. The example is a starting point, not a complete service: networking, secrets, resource reservations, service registration, ACLs, and environment-specific constraints still need review.

Use the plan as a concurrency check, not a deployment lock

Run a plan and inspect which allocations Nomad will replace, whether the canary can be placed, and what resource headroom the scheduler sees:

nomad job plan billing-api.nomad.hcl
nomad job run -check-index <modify-index-from-plan> billing-api.nomad.hcl

The plan’s modify index lets job run -check-index reject a submission if the server-side job changed after planning. It does not freeze the cluster or reserve capacity. Another deployment can still consume resources after the plan; review the actual deployment state after submission rather than treating a successful dry run as a guarantee.

After submitting the candidate, inspect both the job and deployment:

nomad job status billing-api
nomad deployment status <deployment-id>

Look for the expected candidate count, placement state, health, and whether the deployment requires manual promotion. If the canary remains pending or unhealthy, pause at the deployment boundary. Do not promote merely because the allocation process is running.

Configure health gates that answer a specific question

With the default health_check = “checks”, Nomad considers task state and associated service checks when determining allocation health. min_healthy_time requires an allocation to remain healthy for a minimum interval before more allocations are unblocked. healthy_deadline limits how long a new allocation may take to become healthy. progress_deadline starts when the first deployment allocation is created and resets when an allocation becomes healthy or an operator manually promotes the deployment; if neither event occurs before the deadline, the deployment fails.

These are scheduler health signals, not proof that customers can complete a business transaction. A readiness endpoint can pass while authorization, a downstream database, a queue consumer, or a critical query is broken. Define additional, read-only smoke tests and compare labeled error rate, latency, saturation, and business outcomes during the observation window. Avoid a test that creates real orders, charges a payment method, or mutates production data unless that behavior is explicitly isolated and reversible.

Nomad does not decide a canary’s share of HTTP requests merely because an allocation is marked canary. Service discovery and the load balancer or mesh determine what receives traffic. Before the update, verify whether the candidate is registered alongside stable allocations and what routing policy will do with it. If the intent is a small traffic cohort, configure and verify that policy in the routing layer; do not infer a 1-to-4 request ratio from canary = 1.

Promote only after the observation gate passes

With auto_promote = false, the operator explicitly promotes a deployment after checking the canary. Nomad’s deployment watcher can auto-promote only when the deployment’s canaries are placed and healthy and every task group that has canaries is configured for auto-promotion. A job with multiple task groups therefore needs a deliberate review of the combined promotion behavior; a healthy canary in one group is not a reason to assume every group is ready.

For an approved candidate, use the deployment identifier reported by Nomad:

nomad deployment promote <deployment-id>

Promotion allows the rest of the task group’s rolling update to proceed at max_parallel. Keep monitoring until the deployment reaches a terminal successful state and the expected new allocations are healthy. Promotion is not a traffic-weight instruction and does not prove schema compatibility or correctness of external side effects.

If the candidate fails, use the deployment’s failure workflow and inspect the resulting job and allocation state. auto_revert = true asks Nomad to revert to the last stable job on deployment failure; the documentation defines stability in terms of healthy allocations. This is a workload-specification rollback, not a reverse migration. It cannot undo a database write, a message already sent, or a destructive schema change. Use backward-compatible expand-and-contract migrations and an independent recovery plan for persistent state.

Scope and version boundaries

The example is for service jobs. System jobs interpret canary differently: it is a percentage of feasible nodes for destructive updates, and newly registering nodes do not receive placements until promotion. The update documentation also disallows canaries with per-allocation volumes. Do not copy the service-job arithmetic into either case.

Keep the deployment object, its health evidence, routing state, and application-level telemetry together in the release record. The useful result of a canary is not simply “Nomad created one allocation.” It is a documented decision that the staged version was placed, met the configured health contract, received the intended traffic, and passed independent checks before the remainder of the fleet changed.

Related:

Sources:

Comments