Nomad Periodic Jobs: Overlap, Time Zones, and Missed Runs

Nomad periodic jobs look like distributed cron, but their operational contract is narrower than a durable work queue. A periodic specification tells Nomad when to dispatch batch work; it does not guarantee that every scheduled occurrence will wait in a backlog, that two independent jobs will serialize, or that a local wall-clock time exists exactly once on every calendar day. Those distinctions matter when a job rotates credentials, exports customer data, charges an account, or performs any other operation where an unplanned duplicate or omission has consequences.

This guide focuses on periodic batch and sysbatch jobs, which are the job types accepted by Nomad’s periodic block. The current HashiCorp reference prefers the plural crons field; the older singular cron field was deprecated in favor of crons in Nomad 1.6.2. The behavior discussion below is checked against the current online documentation and an immutable Nomad source snapshot at commit bf24b0dd97430d3b364b68b1f7375e85e585778f. Confirm the installed Nomad version and its documentation before relying on release-specific details.

A periodic job is a dispatcher, not a service timer

Nomad’s periodic scheduler evaluates the configured expressions and dispatches a child batch job for a matching launch time. At the pinned source revision, the derived child receives a launch-time-based ID, records its parent, and has its periodic block removed. The parent is the schedule definition; the child is one attempted execution. That separation is useful during investigation: inspect the parent to understand the policy, then inspect child evaluations and allocations to understand what happened for a particular firing.

The following is a periodic-block excerpt for an explicit UTC schedule and same-job overlap policy. It is not a complete jobspec: a valid submitted job also needs its task group and task, which are intentionally omitted here.

periodic {
  crons            = ["0 15 2 * * *"]
  time_zone        = "UTC"
  prohibit_overlap = true
}

In this six-field example, the schedule is evaluated at 02:15:00 UTC each day. Validate the expression with the Nomad version that will run it; do not infer cron syntax from another scheduler. crons can contain more than one expression, and Nomad evaluates the next earliest matching time. For a first review that does not submit the job, use local validation and rendered output:

nomad job validate inventory-snapshot.nomad.hcl
nomad job run -output inventory-snapshot.nomad.hcl

Use those commands on a complete, reviewed test jobspec that contains a harmless task; they are not a validation of the excerpt above. The second command prints parsed JSON; it is not nomad job run inventory-snapshot.nomad.hcl and does not submit a workload. Keep the reviewed file, Nomad version, rendered output, and any plan result together so a later operator can tell which schedule was approved.

`prohibit_overlap` skips an occurrence; it is not a global lock

With prohibit_overlap = true, Nomad prevents an instance of this periodic job from overlapping an earlier instance of the same job. It does not prevent a different periodic job from running at the same time. Shared databases, APIs, storage systems, and external rate limits still need capacity planning or an explicit coordination mechanism across job definitions.

The source implementation makes the missed-run consequence concrete. At the pinned revision, the dispatcher advances the next scheduled launch, checks child evaluations and allocations for nonterminal work, and skips creating an evaluation when a previous child is still active. The code does not put that missed occurrence into a durable backlog for later replay. Treat this as the implementation behavior at that exact commit, not a timeless promise across all Nomad versions. Design the workload to tolerate a skipped interval, or add a separate durable queue/reconciliation mechanism if every unit of work must eventually be processed.

An operator can optionally configure group.max_run_duration with prohibit_overlap; HashiCorp documents that this can terminate an earlier run before a later scheduled run starts. That is a termination policy, not proof that the task rolled back cleanly. Before setting a maximum duration, determine what happens if Nomad stops a task during a partially completed export, migration, or remote write. Prefer idempotent work units, checkpoints, and an explicit recovery procedure to treating a timeout as a correctness guarantee.

The manual nomad job periodic force command is a separate path. The command reference warns that a forced launch can violate prohibit_overlap; a forced request is therefore not a safe generic retry. It also requires suitable ACL capability. Before using it, verify whether the original child is active, whether the operation is idempotent, and who owns any external side effects. An enabled = false periodic block prevents both scheduled and forced launches according to the current job-specification reference; do not confuse disabling dispatch with stopping an already-running child.

Pick a calendar contract deliberately

Nomad evaluates schedules in UTC by default, and HashiCorp recommends expressing periodic schedules with UTC unless local-clock behavior is required. That default is particularly useful for maintenance that must occur at a fixed elapsed-time boundary rather than at “2:15 a.m. wherever the cluster happens to be.” Name the zone explicitly in reviewed jobspecs, even when it is UTC, so operators do not have to infer the intended calendar later.

If a job must follow a named local zone, account for daylight-saving transitions. Nomad’s documentation says a schedule in the skipped spring-forward hour is skipped for that day, while a schedule in the repeated fall-back hour runs twice. A daily local-time export can therefore have one missing day or a duplicate execution; an expression does not smooth over the discontinuity. If either outcome is unacceptable, use UTC and translate business dates in the task, or make the operation explicitly detect already-completed work by business key.

For example, suppose a report is legally defined as “the previous local business day.” The scheduler can dispatch at a stable UTC time, while the application computes the target business date in the named jurisdiction and stores that date as an idempotency key. If the dispatcher skips one occurrence, the next execution can reconcile unprocessed dates. If the repeated hour fires twice, the unique business-date key prevents a second report from being committed. This design separates a scheduler’s launch signal from the application’s definition of completed work.

Observe parent, child, and side effect separately

For a scheduled run, capture at least the parent job ID, child job ID, evaluation ID, allocation state, scheduled or launch time, and the task’s own result identifier. Nomad’s derived child IDs include the launch timestamp at the cited source revision, but external effects should still carry their own idempotency key; a job ID is not a transaction identifier for a payment gateway or database.

A read-only operational review can inspect the schedule parent and its children with nomad job status and nomad job inspect for the parent and relevant child IDs. Review task logs and allocation state before deciding whether to retry. A terminal failed allocation does not establish that an external write failed: the remote service may have accepted the request before the client lost its response. Reconcile against the authoritative side-effect system before resubmitting non-idempotent work.

Use a staging namespace to test three distinct cases: a normal firing that completes before the next interval, a deliberately long-running instance with overlap prohibited, and a manually forced run while that instance remains active. The third case demonstrates why force is a privileged exception rather than routine recovery. Keep destructive side effects disabled in this test; use a disposable destination and assert the number of external records, not only the number of Nomad allocations.

Periodic scheduling is only the time trigger. Kubernetes CronJobs and AWS EventBridge Scheduler have their own concurrency, missed-start, retry, and time-zone semantics; a migration between them should translate the operational contract, not just copy a cron expression. Document who is responsible for skipped work, how duplicates are made harmless, and how an operator can distinguish “not launched” from “launched but outcome unknown.”

Related:

Sources:

A periodic job is a dispatcher, not a service timer

prohibit_overlap skips an occurrence; it is not a global lock

Pick a calendar contract deliberately

Observe parent, child, and side effect separately

Comments

`prohibit_overlap` skips an occurrence; it is not a global lock